Emotion support dialogue strategy management method fusing user feedback

Abstract

The invention provides an emotional support dialogue strategy management method fusing user feedback, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining input dialogue information; inputting the dialogue information into a pre-trained knowledge enhancement encoder, and outputting context representation of enhanced knowledge; based on the context representation, selecting to generate a reply strategy for pacifying the user; generating a candidate reply consistent with the reply strategy based on the reply strategy and the context representation; and predicting feedback of the user to the candidate replies, and iteratively generating an optimal reply strategy based on the feedback. According to the method, the adaptive reply strategy can be effectively and accurately predicted, and the reply consistent with the reply strategy can be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an emotion support dialogue strategy management method integrating user feedback. Background Art

[0002] With the development of the economy and the intensification of competition, people's pace of life and work is getting faster and faster. For people in the economic transition period, the psychological pressure is undoubtedly increased. In order to relieve emotional pressure, the emotional support dialogue system came into being. It aims to understand the emotional distress of human users and provide emotional companionship in the form of dialogue by providing empathetic responses and counseling suggestions. This dialogue system not only focuses on understanding and responding to the user's emotional state, but also helps users deal with emotional problems and enhance their emotional well-being by providing comfort, encouragement and support. In order to better solve user problems, the system needs to adjust the dialogue strategy in a timely manner, that is, to achieve effective strategy management and ensure that the generated responses are consistent with the selected strategy.

[0003] In the research of emotional support dialogue tasks, the current mainstream methods can be summarized into three categories. The first type of method relies on retrieval technology, which generates replies by retrieving text fragments that match the context from a pre-built corpus. Although this type of method can provide emotional support to a certain extent, its limitation lies in the lack of a dynamic adjustment mechanism, which makes it difficult to optimize the reply strategy based on user feedback, resulting in the generated replies failing to fully meet the user's emotional needs. The second type of method uses neural networks to directly generate dialogue content, but there are often problems of inaccuracy and ambiguity in emotional expression, especially in the case of insufficient data sets or complex emotions, generating replies that are difficult to achieve the expected effect of emotional support. The third type of method is knowledge-enhanced generation technology, which enriches the generated dialogue content by introducing knowledge graphs, parameterized knowledge or unstructured text. This type of method improves the richness and consistency of the dialogue to a certain extent, but still faces the problem of the generated content not being consistent with the facts, and relies on the quality of external knowledge, which easily leads to insufficient emotional expression. Therefore, how to accurately predict the adapted reply strategy and generate replies consistent with the reply strategy is an extremely important technical problem to be solved. Summary of the invention

[0004] In order to solve the problem that the above-mentioned prior art is difficult to accurately predict the adapted reply strategy and generate a reply consistent with the reply strategy, the present invention proposes an emotion support dialogue strategy management method integrating user feedback, which can effectively and accurately predict the adapted reply strategy and generate a reply consistent with the reply strategy.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] A method for managing emotional support dialogue strategies integrating user feedback includes the following steps:

[0007] S1. Get the input dialogue information;

[0008] S2. inputting the dialogue information into a pre-trained knowledge enhancement encoder, and outputting a contextual representation of the enhanced knowledge;

[0009] S3. Based on the context representation, select and generate a reply strategy to appease the user;

[0010] S4. Based on the reply strategy and the context representation, generating a candidate reply consistent with the reply strategy;

[0011] S5. Predict user feedback on the candidate responses, and iteratively generate an optimal response strategy based on the feedback.

[0012] Preferably, constructing the knowledge enhancement encoder includes: injecting psychological knowledge into a preset language model PLM in the form of an adapter to obtain the knowledge enhancement encoder.

[0013] Preferably, the selecting and generating a reply strategy for comforting the user based on the context representation includes:

[0014] Construct a high-level state graph representing the relationship between the context representation and the reply strategy, wherein the high-level state graph comprises a node V and an edge E, wherein the node V is represented as a set of utterance representations. and strategy set representation s, the edge E includes the connection between two consecutive utterances, and the connection between the supporter's utterance and its specific strategy;

[0015] A number of stacked graph convolutional network layers are used to model the correlation between the speeches and strategies involved in the high-level state graph, and a graph representation G of the high-level state graph is obtained. * ;

[0016] The representation of the high-level state diagram is captured using a neural network based on a self-attention mechanism, and the expression used to assist the high-level state representation is obtained as follows:

[0017]

[0018] in, is the graph representation at timestamp t, is the tth high-level state representation, which is used to assist the policy selection network and the policy evaluation network in generating a response policy.

[0019] Preferably, the strategy selection network generates the following expression for the response strategy based on the high-level state representation:

[0020]

[0021] in, represents the high-level action, i.e., the response strategy, tanh represents the first activation function, represents the first weight vector, Represents the first bias parameter.

[0022] Preferably, the strategy evaluates the network input high-level state and high-level actions The expression for predicting the current Q value is as follows:

[0023]

[0024] Among them, ReLU represents the second activation function, represents the second weight vector, represents the third weight vector, Represents the second bias parameter.

[0025] Preferably, the generating of candidate replies consistent with the reply strategy based on the reply strategy and the context representation is as follows:

[0026]

[0027] in, It represents the action of expressing the bottom layer, that is, the generation of candidate responses, and Decoder represents the decoder. Contextual representations that represent psychology-based knowledge, Represents a reply strategy embedding.

[0028] Preferably, S5 specifically includes: using a preset reward model to predict the user's feedback on the candidate reply, if the feedback is positive, then outputting the candidate reply as the optimal reply strategy; if the feedback is negative, then using a hierarchical reinforcement learning method to iteratively generate the optimal reply strategy.

[0029] Preferably, the calculation expression of the loss function used in training the reward model is:

[0030]

[0031] Among them, loss(.) represents the loss function, r θ (q,p i ) represents the reward model for mental health problems q and responses p i scalar output, θ represents the model parameters, p i ∈(p 0 ,p 1 ), i means reply p i, D represents the dataset with human ranking, and σ represents the standard deviation.

[0032] The present invention proposes an emotion support dialogue strategy management system integrating user feedback, comprising:

[0033] Input module, used for inputting dialogue information;

[0034] A knowledge enhancement encoder, configured to receive the dialogue information and output a contextual representation of the enhanced knowledge;

[0035] a reply strategy selection unit, configured to select and generate a reply strategy to appease the user according to the context representation;

[0036] a strategy consistent generation unit, configured to generate a candidate reply consistent with the reply strategy according to the reply strategy and the context representation;

[0037] The user feedback prediction unit is used to predict the user's feedback on the candidate reply and iteratively generate the optimal reply strategy based on the feedback.

[0038] Preferably, the iterative generation of the optimal reply strategy based on the feedback includes: using a preset reward model to predict the user's feedback on the candidate reply, if the feedback is positive, then the candidate reply is finally output as the optimal reply strategy; if the feedback is negative, then a hierarchical reinforcement learning method is used to iteratively generate the optimal reply strategy again.

[0039] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0040] The present invention proposes an emotion support dialogue strategy management method integrating user feedback. Firstly, a knowledge enhancement encoder is used to encode the input dialogue information, so as to better encode the context and accurately perceive the user's psychological state. Secondly, a response strategy is selected and generated according to the context representation, so as to find a response strategy that can comfort the user. Then, according to the context representation and the response strategy, the corresponding semantics and language expression details are captured to generate candidate responses consistent with the response strategy. Finally, the user's feedback on the candidate responses is predicted, and the optimal response strategy is generated in an iterative manner, so as to learn the strategy selection at the cognitive level and the strategy consistent response generation process at the expression level, thereby effectively and accurately predicting the adapted response strategy and generating a response consistent with the response strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Example diagram representing the value of existing user feedback;

[0042] Figure 2 Example diagram representing the value of existing psychological knowledge;

[0043] Figure 3 A flowchart of a method for managing an emotion support dialogue strategy integrating user feedback proposed in an embodiment of the present invention is shown;

[0044] Figure 4 A flowchart showing a method for managing an emotion support dialogue strategy integrating user feedback proposed in an embodiment of the present invention;

[0045] Figure 5 A structural block diagram of an emotion support dialogue strategy management system integrating user feedback proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The accompanying drawings are only used for illustrative purposes and should not be construed as limiting the present embodiment;

[0047] It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings;

[0048] In order to facilitate understanding of this embodiment, first, the prior art information of this embodiment is introduced as follows:

[0049] Due to insufficient understanding of response strategies, existing methods often have a mismatch between the predicted strategy and the actual response strategy. For example, the system predicts a strategy of "asking questions", such as asking "Why are you sad?", but the generated response is not very expressive, such as "sadness is a common type of emotion". This kind of response similar to the "information" strategy is difficult to achieve the effect of relieving user emotions.

[0050] User feedback plays a vital role in the emotion-support dialogue system, which not only measures the user's satisfaction with the generated responses, but also directly affects the direction and content of the dialogue. Figure 1 As shown in the figure, feedback can help the machine more accurately understand the problems that cause users' negative emotions, thereby optimizing the response method. For example, when a user faces unemployment difficulties, if there is a lack of feedback, the machine may give some statements that do not fit the user's actual situation, making it difficult to alleviate the user's confusion and anxiety. Taking into account user feedback, the machine can better understand the specific difficulties faced by the user by guiding the user to share specific difficulties. Based on these feedbacks, the dialogue system can more accurately grasp the user's needs and provide users with more accurate and practical personalized suggestions, such as recommending positions in specific industries or providing targeted training programs. This can make users feel understood and more confident to deal with difficulties, soothing their emotions. However, it is not easy to accurately perceive and understand user feedback. In addition, how to use feedback to adjust dialogue strategies and solve problems is also very challenging.

[0051] In addition, mental health knowledge plays a vital role in emotional support dialogue systems. It can help the system understand the user's emotional state and psychological needs more accurately, thereby providing more targeted suggestions and problem solutions. However, current research has neglected the introduction of mental health knowledge, resulting in a lack of professionalism in the generated content. Figure 2 As shown in the figure, the machine only relies on simple matching to retrieve relevant text, which lacks specificity, resulting in the generated suggestions being empty and unable to soothe the user's confusion. In addition, the retrieval process may introduce noise information that does not match the user's current situation or is outdated, further reducing the effectiveness of the suggestions. By introducing parameterized mental health knowledge, the machine can understand the user's emotional state more deeply and provide specific and personalized suggestions. Therefore, the introduction of psychological knowledge can effectively make up for the limitations of text-based retrieval, improve the quality of replies, and enhance users' trust and satisfaction with the results.

[0052] In order to solve the above problems, this embodiment proposes a dialogue strategy management framework that integrates user feedback. The framework can accurately perceive the user's emotional state based on context and psychological knowledge, and then select appropriate strategies to generate appropriate responses.

[0053] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0054] Example 1

[0055] This embodiment proposes a method for managing emotional support dialogue strategies that integrate user feedback. The emotional support dialogue aims to generate supportive responses through carefully designed strategies, thereby helping the dialogue system to alleviate the user's negative emotions. 1 ,…,u t ], and the output is a supportive response y with a specific strategy. This embodiment adopts a hierarchical reinforcement learning method and regards the emotional support dialogue as a two-level task including a strategy level and an expression level. The user is regarded as the environment, and the model is responsible for completing the two-level task. First, the high-level (i.e., strategy level) task is based on the context C = [u 1 ,…,u t ]Select a response strategy h . In the selected response strategy a h After observing the context, the low-level (i.e., expression-level) task is to generate candidate responses a that are consistent with the strategy. l . The candidate responses are then evaluated based on user feedback and treated as reward signals from the environment to train the reward model. The concepts involved are as follows:

[0056] High-level states h :s h Contains the conversation history and the collection of strategies employed in the discourse of supporters.

[0057] High-level action h : The high-level action refers to the selection of a response strategy. h is considered as the selected strategy.

[0058] High-level rewards h : The high-level reward is a scalar used to evaluate the selected policy, measured by the accuracy of the selected policy and the feedback of the corresponding generated response.

[0059] Low-level states l :s l Contains the conversation history, where each supporter's speech is not labeled with a policy label, and contains policy constraints.

[0060] Low-level action l : The low-level action refers to generating a response. l is considered as the response generated.

[0061] Low-level rewards l : The low-level reward is a scalar used to evaluate user feedback.

[0062] like Figure 3 and Figure 4 As shown, the method comprises the following steps:

[0063] S1. Get the input dialogue information;

[0064] S2. inputting the dialogue information into a pre-trained knowledge enhancement encoder, and outputting a contextual representation of the enhanced knowledge;

[0065] In S2, professional psychological knowledge is introduced to enhance the context representation, so as to better understand user problems, comfort users and provide more reliable solutions. However, when integrating psychological knowledge, relying on the corpus to retrieve similar texts may introduce noise, resulting in the retrieval of irrelevant or non-user-required content, thereby affecting performance. In addition, if fine-tuning is used to inject new knowledge or knowledge base into the machine, large-scale retraining is required. Therefore, constructing the knowledge enhancement encoder includes: injecting psychological knowledge into a preset language model PLM in the form of an adapter to obtain the knowledge enhancement encoder; avoiding large-scale training. The adapter is a lightweight model component that can combine a pre-trained general model with knowledge in a specific field to improve the performance of the model in a specific field. This step uses a plug-and-play paradigm for knowledge injection, and injects the knowledge base into the frozen language model through a knowledge plug-in, i.e., an adapter. Specifically, a knowledge adapter KA is trained, and the pre-trained language model PLM is enhanced by psychological knowledge. Given a pre-trained language model PLM as the backbone, psychological knowledge K is injected into P by training KA and freezing the parameters of PLM. Among them, K is the question-and-answer corpus MentalQA about mental health issues. During the training process, the Masked Language Model task is used to train KA. In order to ensure that the adapter not only learns language skills but also can master psychological knowledge, the method of randomly masking non-stop words is adopted. After the training is completed, KA can inject psychological knowledge into the language model in a plug-and-play manner. Then the context representation is enriched using the knowledge adapter for psychological knowledge injection. For example, given u t Represents the speech at timestamp t, and obtains Indicates speech t The hidden vector of Indicates speech t The utterance representation of the pre-trained language model and the Encoder represent the encoder. This can combine the versatility of the pre-trained language model with the domain expertise of the adapter, effectively improving the performance of the dialogue system in the field of mental health without significantly increasing additional parameters and complexity. In addition, the adapter is optimized for specific fields and can better adapt to the context and needs of the mental health field and provide more accurate and professional responses. By leveraging the versatility of the pre-trained language model and the adapter to model the expertise in the field of mental health, the performance of the dialogue system in the field of mental health can be effectively improved without adding too many additional parameters or complexity. At the same time, because the adapter is an optimized model for a specific field, it can better adapt to the context and needs of the mental health field and provide more effective responses.

[0066] S3. Based on the context representation, select and generate a reply strategy to appease the user;

[0067] In S3, based on the context representation, a reply strategy to appease the user is selected. In order to better learn the relationship between the reply strategy and the context, a graph-based method is designed, including:

[0068] A high-level state graph is constructed to represent the relationship between the context representation and the reply strategy. The high-level state graph includes nodes V and edges E. The user's speech, the supporter's speech, and the strategy in the supporter's speech are all represented as nodes. Therefore, the node V is represented as a set of discourse representations. and a policy set representation s, for example, The definition graph contains n nodes, which are represented by V = v i ,1≤i≤n. The edge E includes the connection between two consecutive utterances and the connection between the supporter's utterance and its specific strategy; (1) The connection between two consecutive utterances, that is, This connection maintains the coherence and order of the dialogue. (2) The connection between the supporter’s speech and his specific strategy, i.e. u →s. This connection identifies the strategy adopted by the supporter to reply. By constructing a state graph of this strategy layer, it is easy to grasp the relationship between the reply strategy and the context. The state of the strategy layer is learned by using a graph construction method, which aims to capture the pattern of strategy selection.

[0069] Using L G A stacked graph convolutional network layer (GCN) is used to model the correlation between the speech and strategy involved in the high-level state graph, and the graph representation G of the high-level state graph is obtained. * , aiming to obtain the policy layer state representation. Since the high-level state graph is initialized by the knowledge adapter, the representation of node v in the g+1th layer is updated according to the proposed method:

[0070]

[0071] in, is the set of direct neighbors of node v, and Λ is the normalized matrix. are two learnable weight matrices of the g-th layer. G After stacking graph convolutional network layers (GCN), we get the graph representation G * .

[0072] The representation of the high-level state diagram is captured using a neural network Transformer based on a self-attention mechanism. By processing the input sequence with the Transformer layer, the expression used to assist the high-level state representation is obtained as follows:

[0073]

[0074] in, is the graph representation at timestamp t, is the tth high-level state representation, which is used to assist the policy selection network and the policy evaluation network in generating a response policy.

[0075] The strategy selection network generates a response strategy based on the high-level state representation. Specifically, The input to the fully connected layer to generate the expression of the t-th high-level action is as follows:

[0076]

[0077] in, represents the high-level action, i.e., the response strategy, tanh represents the first activation function, represents the first weight vector, represents the first bias parameter;

[0078] The goal of the policy evaluation network is to calculate the Q value, which represents the total reward after selecting a certain policy. The high-level Q value is also a judgment on whether the policy selected by the policy selector matches the current high-level state, formalized as:

[0079]

[0080] in is the high-level reward after selecting a strategy, which includes two important aspects: the consistency between the expected reply strategy and the actual adopted strategy, and the user's feedback on the reply generated by the selected strategy. γ∈[0,1] is a discount factor that takes into account the position deviation effect in practical applications and ensures that the strategy can obtain higher rewards. In this case, the strategy evaluator is able to capture the user's preference for the strategy level at each position.

[0081] Specifically, a fully connected layer is used to estimate the Q value. At the tth position, the strategy evaluates the high-level state of the network input and high-level actions The expression for predicting the current Q value is as follows:

[0082]

[0083] Among them, ReLU represents the second activation function, represents the second weight vector, represents the third weight vector, Denotes the second bias parameter. Based on the expected Q value, the strategy selector can update its corresponding parameters to improve performance.

[0084] S4. Based on the reply strategy and the context representation, generating a candidate reply consistent with the reply strategy;

[0085] In S4,

[0086] To generate strategy-consistent responses, this unit aims to enhance the context representation and strategy constraints based on psychological knowledge. Generating candidate responses. We first infer a policy vector and then generate responses consistent with the selected policy.

[0087] In order to avoid the situation where the selected strategy and the response content are inconsistent, that is, the words do not express the meaning, a potential diffusion model is used to learn the strategy concept. Its goal is to encode the strategy concept into the intermediate representation of the pre-trained model and use the semantic prior to guide the language expression of the concept. In order to capture the unique expression and language structure related to the strategy, the strategy vector is introduced into the embedding space of the pre-trained language model. Specifically, a new pseudo-word a is used * to represent the strategy we want to learn. This pseudo-word is treated as a normal word and is used to generate responses consistent with the corresponding strategy. During the optimization process, a set of examples showing the dialogue strategy in different scenarios is input, and neutral text of the template is randomly sampled as conditional input, similar to "a * Examples of "or" * The optimization goal is as follows:

[0088]

[0089] Among them, c θ (p) represents the model that maps the conditional input p to the conditional vector, t represents the time step, and z t represents the potential noise at time t, ∈ is the unscaled noise sample, ∈ θ is a denoising network. The goal of this task is to correctly remove the noise added to the latent representation of the policy response. It is expected that the embedding vector learned through this process can capture the unique expression and language structure of the policy. By providing the diffusion model with a specific response policy Example of getting the reply strategy embedding Used to generate custom responses that match this response strategy. Finally, consider the knowledge-rich contextual representation (where C represents the context) to generate candidate replies. In order to generate replies consistent with the selected reply strategy, the reply strategy embeds It is treated as a regular word embedding and generates candidate responses consistent with the response strategy as follows:

[0090]

[0091] in, It represents the action of expressing the bottom layer, that is, the generation of candidate responses, and Decoder represents the decoder. Contextual representations that represent psychology-based knowledge, represents a reply strategy embedding. By treating a reply strategy as a new pseudo-word in the model vocabulary, the reply strategy can be used to guide the language generation process.

[0092] S5. Predict user feedback on the candidate responses, and iteratively generate an optimal response strategy based on the feedback.

[0093] In this embodiment, an encoder incorporating psychological knowledge is first constructed to perceive the user's psychological state. Through the knowledge adapter, the encoder can parameterize these psychological knowledge and reduce the interference of noise in the retrieval process. Considering that the graph structure is easier to capture the relationship between context and reply strategy, this embodiment constructs a strategy selection unit based on the graph structure to select the best comforting strategy from the strategy pool according to the perceived user state. Subsequently, this embodiment adopts the potential diffusion model to learn the strategy embedding, capture the corresponding semantics and language expression details, and thus generate a reply consistent with the strategy. Finally, this embodiment designs a training model for the hierarchical reinforcement learning method, which uses a reward model to evaluate the user's feedback on potential replies and finds the optimal strategy and the corresponding reply in an iterative manner. The experimental results on the benchmark dataset show that the performance of this method is superior. The method proposed in this embodiment is of great significance in practical applications, especially in the fields of mental health care, emotional support dialogue, and psychological services for the elderly. By introducing a parameter knowledge enhancement framework driven by user feedback, this method can perceive the user's emotional state in real time and dynamically adjust the reply strategy, thereby providing more personalized, targeted and effective emotional support. This technology based on psychological knowledge injection and strategy consistency generation can improve the machine's ability to understand and respond to user needs in emotionally complex and changeable dialogue scenarios. Therefore, this method can effectively improve the intelligence and practicality of the dialogue system and promote the application and development of intelligent dialogue technology in mental health services.

[0094] It should also be noted that in the face of emotional support dialogue tasks, traditional methods usually retrieve text fragments that match the context from the corpus to generate replies. However, these methods ignore user feedback and cannot dynamically adjust the reply strategy according to the user's emotional state. Existing methods are suitable for generating relatively fixed replies, but it is difficult to effectively alleviate the user's negative emotions. In addition, these methods are prone to introduce noise during the retrieval process, resulting in the generated replies not accurately matching user needs. In order to address the above shortcomings, this embodiment proposes a new framework that breaks through the limitations of traditional emotional support dialogue systems and effectively improves the personalization of dialogues and user satisfaction. Specifically, this embodiment constructs a hierarchical reinforcement learning model based on user feedback by injecting professional knowledge in the field of mental health into a pre-trained language model in the form of an adapter. Although traditional methods have been applied in emotion generation tasks, they often cannot handle complex psychological problems. This embodiment combines mental health knowledge with user feedback to dynamically adjust the reply strategy and generate replies consistent with the strategy. In addition, this embodiment also uses a potential diffusion model to learn strategy concepts to ensure that the content of the generated reply conforms to the established dialogue strategy. The advantages of the method proposed in this embodiment include:

[0095] This paper proposes an emotional dialogue framework based on hierarchical reinforcement learning, which can adjust strategies based on user feedback to generate personalized soothing responses and improve user experience.

[0096] By using an adapter to inject mental health knowledge into the pre-trained language model, we can better understand the user's mental state and generate personalized recommendations based on the user's specific situation, significantly reducing the noise problem in traditional methods.

[0097] The strategy embedding is learned through the latent diffusion model to ensure that the generated response content is consistent with the selected strategy, and the feedback estimation unit ensures that the generated response meets the user's expectations in terms of logic and common sense.

[0098] In addition, in order to further evaluate the performance of the emotion support dialogue strategy management method integrating user feedback proposed in this embodiment, this embodiment uses the ESConv dataset in the experiment, which is widely used in emotion support dialogue tasks. For the convenience of analysis, this embodiment adopts two evaluation methods: automatic evaluation indicators and manual evaluation indicators. The automatic evaluation indicators used in the experiment include PPL, BLEU, DIST, ROUGE-L, Meteor and CIDEr, which are widely used in dialogue generation tasks. Among them, PPL (Perplexity) is used to measure the effectiveness of the language model. The lower the score, the better the model performance; BLEU analyzes the co-occurrence of generated responses and real texts on n-grams; DIST measures the diversity of the corpus by calculating the ratio of all unique n-grams in the output; ROUGE-L (Longest Common Subsequence) evaluates the accuracy of the generated text based on the longest common subsequence between the candidate sentence and the target sentence, and Meteor is used to evaluate the precision and recall of the generated text; CIDEr measures the quality of the response by calculating the similarity of n-grams between the generated text and the reference text. The manual evaluation part recruits participants to subjectively evaluate the quality of generated responses, using evaluation dimensions including fluency (Flu), identification (Ide), comfort (Com), suggestion (Sug), and overall performance (Ove). Participants are required to compare the response performance of two different models on these dimensions based on the generation of each round of dialogue. Experimental results show that this method is significantly better than traditional methods.

[0099] Example 2

[0100] This embodiment further explains S5. After the modeling expression layer generates candidate replies with consistent reply strategies, the user's feedback on the candidate replies is evaluated, and the user's feedback on the candidate replies is predicted using a preset reward model to determine whether the reply effectively interacts with the user, and then used as the final output; if the feedback is positive, the candidate reply is used as the final output of the optimal reply strategy; if the feedback is negative, the hierarchical reinforcement learning method is used to iteratively generate the optimal reply strategy.

[0101] However, there is a lack of user feedback in the benchmark dataset because the information provided by the data is incomplete. Therefore, it is proposed to learn user feedback by fine-tuning the reward model. This reward model takes the context and the reply as input, and then outputs a score for the reply. To train the reward model, a pre-trained language model and a randomly initialized linear layer are used. Drawing on Stiennon's ideas in the paper Learning to summarize with human feedback, the reward model is based on a mental health knowledge base MentalQA, which contains the rankings of multiple responses of users to the same mental health question. Specifically, the ranking is obtained through the user's like behavior, avoiding personal bias. Given a mental health question q, the reward model predicts which reply p∈(p 0 ,p 1 ) is better. When p i When it is preferred, the loss function used in training the reward model is calculated as:

[0102]

[0103] Among them, loss(.) represents the loss function, r θ (q,p i ) represents the reward model for mental health problems q and responses p i scalar output, θ represents the model parameters, p i ∈(p 0 ,p 1 ), i means reply p i , D represents the dataset with human rankings, and σ represents the standard deviation. At the end of training, the output of the reward model is normalized so that the average score of all responses in the dataset is 0.

[0104] The hierarchical reinforcement learning method is used to iteratively generate the optimal response strategy. The specific process is as follows:

[0105] To ensure fast and stable convergence of hierarchical reinforcement learning, a multi-faceted objective is designed to jointly model the loss functions of the two levels. In each dialogue turn, a state graph is first constructed, and a reply strategy is selected based on the graph representation.

[0106] Specifically, the policy executor is responsible for selecting the response strategy, while the policy evaluator calculates the Q value. The classic mean square error loss function is used to train them:

[0107]

[0108] is the target Q-value at time point t, which consists of the current reward and the future Q-value from the target network. h and θh′ is the set of parameters of the policy evaluator, the difference is that θ h During the training process, θ is updated h′ is the set of prior empirical parameters that is fixed during optimization. Indicates the target policy of the policy enforcer.

[0109] The Q-values ​​generated by the policy evaluator are used to update the policy executor via policy gradients. By maximizing the overall ranked expected reward, actions with higher Q-values ​​are generated:

[0110]

[0111] where a h =μ h (s h ∣φ h ) represents the response strategy selected by the policy executor through its policy. Finally, the losses of the lower-level agents are aggregated as follows:

[0112] L h =L(θ h )+L(φ h ).

[0113] Next, psychological knowledge is integrated into the low-level state representation through a knowledge adapter. The adapter is trained with a masked language model (MLM) loss. Based on the selected strategy and knowledge-enhanced representation, the reply policy embedding is used to generate a reply consistent with the strategy:

[0114]

[0115] is the low-level target Q value at time t, θ represents the low-level reward, which consists of the current reward and the similarity between the generated response and the reference response. l and θ l′ is the set of parameters of the low-level evaluator. l During the training process, θ is updated l′ is the set of prior experience parameters that is fixed during optimization. The loss generated by low-level consensus is:

[0116]

[0117] where μ l refers to the low-level response generation model, a l =μ l (s l ∣φ l ) indicates that the low-level action executor generates a policy-consistent response through the language model.

[0118] Then the loss L(θ generated by the replyl ) and the loss L(φ) for evaluating the reply feedback l )Sum:

[0119] L l =L(θ l )+L(φ l ).

[0120] Finally, the candidate responses are evaluated by estimating user feedback, and the strategy and response are adjusted according to the reward. The estimated loss is as described above.

[0121] Example 3

[0122] See also Figure 5 , this embodiment proposes an emotion support dialogue strategy management system integrating user feedback, including:

[0123] Input module, used for inputting dialogue information;

[0124] A knowledge enhancement encoder, configured to receive the dialogue information and output a contextual representation of the enhanced knowledge;

[0125] a reply strategy selection unit, configured to select and generate a reply strategy to appease the user according to the context representation;

[0126] a strategy consistent generation unit, configured to generate a candidate reply consistent with the reply strategy according to the reply strategy and the context representation;

[0127] The user feedback prediction unit is used to predict the user's feedback on the candidate reply and iteratively generate the optimal reply strategy based on the feedback.

[0128] The iterative generation of the optimal reply strategy based on the feedback includes: using a preset reward model to predict the user's feedback on the candidate reply, if the feedback is positive, then the candidate reply is finally output as the optimal reply strategy; if the feedback is negative, then a hierarchical reinforcement learning method is used to iteratively generate the optimal reply strategy again.

[0129] In this embodiment, the input dialogue information is first encoded using a knowledge-enhanced encoder, with the goal of better encoding the context and accurately perceiving the user's psychological state; secondly, based on the context representation, a response strategy is selected and generated to facilitate finding a response strategy that can appease the user; then, based on the context representation and the response strategy, the corresponding semantics and language expression details are captured to generate candidate responses consistent with the response strategy; finally, the user's feedback on the candidate responses is predicted, and the optimal response strategy is generated in an iterative manner, so as to facilitate learning the strategy selection at the cognitive level and the strategy-consistent response generation process at the expression level, thereby effectively and accurately predicting the adapted response strategy and generating a response consistent with the response strategy.

[0130] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the implementation methods of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for managing emotional support dialogue strategies integrating user feedback, characterized in that: The following steps are involved: S1. Get the input dialogue information; S2. inputting the dialogue information into a pre-trained knowledge enhancement encoder, and outputting a contextual representation of the enhanced knowledge; S3. Based on the context representation, select and generate a reply strategy to appease the user; S4. Based on the reply strategy and the context representation, generating a candidate reply consistent with the reply strategy; S5. Predict user feedback on the candidate responses, and iteratively generate an optimal response strategy based on the feedback.

2. The method for managing emotional support dialogue strategies integrating user feedback according to claim 1, characterized in that: Constructing the knowledge enhancement encoder includes: injecting psychological knowledge into a preset language model PLM in the form of an adapter to obtain the knowledge enhancement encoder.

3. The method for managing emotional support dialogue strategies integrating user feedback according to claim 2, characterized in that: The step of selecting and generating a reply strategy to appease the user based on the context representation includes: Construct a high-level state graph representing the relationship between the context representation and the reply strategy, wherein the high-level state graph comprises a node V and an edge e, wherein the node V is represented as a set of utterance representations and strategy set representation s, the edge E includes the connection between two consecutive utterances, and the connection between the supporter's utterance and its specific strategy; A number of stacked graph convolutional network layers are used to model the correlation between the speeches and strategies involved in the high-level state graph, and a graph representation G of the high-level state graph is obtained. * ; The representation of the high-level state diagram is captured using a neural network based on a self-attention mechanism, and the expression used to assist the high-level state representation is obtained as follows: in, is the graph representation at timestamp t, is the tth high-level state representation, which is used to assist the policy selection network and the policy evaluation network in generating a response policy.

4. The method for managing emotional support dialogue strategies integrating user feedback according to claim 3 is characterized in that: The expression of the response strategy generated by the strategy selection network according to the high-level state representation is as follows: in, represents the high-level action, i.e., the response strategy, tanh represents the first activation function, represents the first weight vector, Represents the first bias parameter.

5. The method for managing emotional support dialogue strategies integrating user feedback according to claim 4, characterized in that: The strategy evaluates the high-level state of the network input and high-level actions The expression for predicting the current Q value is as follows: Among them, ReLU represents the second activation function, represents the second weight vector, represents the third weight vector, Represents the second bias parameter.

6. The method for managing emotional support dialogue strategies integrating user feedback according to claim 5, characterized in that: Based on the reply strategy and the context representation, generating a candidate reply consistent with the reply strategy is as follows: in, It represents the action of expressing the bottom layer, that is, the generation of candidate responses, and Decoder represents the decoder. Contextual representations that represent psychology-based knowledge, Represents a reply strategy embedding.

7. The method for managing emotional support dialogue strategies integrating user feedback according to claim 6, characterized in that: S5 specifically includes: using a preset reward model to predict the user's feedback on the candidate reply, if the feedback is positive, then outputting the candidate reply as the optimal reply strategy; if the feedback is negative, then using a hierarchical reinforcement learning method to iteratively generate the optimal reply strategy.

8. The method for managing emotional support dialogue strategies integrating user feedback according to claim 7, characterized in that: The calculation expression of the loss function used in training the reward model is: Among them, loss(.) represents the loss function, r θ (q,p i ) represents the reward model for mental health problems q and responses p i scalar output, θ represents the model parameters, p i ∈(p0,p1), i represents the reply p i , D represents the dataset with human ranking, and σ represents the standard deviation.

9. An emotional support dialogue strategy management system integrating user feedback, characterized in that: include: Input module, used for inputting dialogue information; A knowledge enhancement encoder, configured to receive the dialogue information and output a contextual representation of the enhanced knowledge; a reply strategy selection unit, configured to select and generate a reply strategy to appease the user according to the context representation; a strategy consistent generation unit, configured to generate a candidate reply consistent with the reply strategy according to the reply strategy and the context representation; The user feedback prediction unit is used to predict the user's feedback on the candidate reply and iteratively generate the optimal reply strategy based on the feedback.

10. The emotion-support dialogue strategy management system integrating user feedback according to claim 9, wherein the iterative generation of the optimal response strategy based on the feedback comprises: Using a preset reward model to predict the user's feedback on the candidate reply, if the feedback is positive, the candidate reply is finally output as the optimal reply strategy; If the feedback is negative, a hierarchical reinforcement learning method is used to iteratively generate the optimal response strategy.

Images