An emotional support dialogue strategy management method fusing user feedback

Abstract

This invention proposes a method for managing emotion-supported dialogue strategies that integrates user feedback, relating to the field of natural language processing. The method includes: acquiring input dialogue information; inputting the dialogue information into a pre-trained knowledge-enhanced encoder, outputting a contextual representation of the enhanced knowledge; selecting and generating a response strategy to appease the user based on the contextual representation; generating candidate responses consistent with the response strategy based on the response strategy and the contextual representation; predicting the user's feedback to the candidate responses; and iteratively generating an optimal response strategy based on the feedback. This invention can effectively and accurately predict suitable response strategies and generate responses consistent with those strategies.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a method for managing emotion-supported dialogue strategies that integrates user feedback. Background Technology

[0002] With economic development and intensified competition, the pace of life and work is accelerating. This undoubtedly increases psychological pressure for people in a period of economic transition. To alleviate emotional stress, emotional support dialogue systems have emerged. These systems aim to understand the emotional distress of human users and provide emotional support through dialogue, offering empathetic responses and guidance. These systems not only focus on understanding and responding to users' emotional states but also help them manage emotional issues by providing comfort, encouragement, and support, thereby enhancing their emotional well-being. To better address user problems, the system needs to adjust its dialogue strategies promptly, achieving effective strategy management and ensuring that the generated responses are consistent with the selected strategies.

[0003] In research on emotion-supporting dialogue tasks, current mainstream methods can be summarized into three categories. The first category relies on retrieval techniques, generating responses by retrieving text fragments from a pre-built corpus that match the context. While this type of method can provide some emotional support, its limitation lies in the lack of a dynamic adjustment mechanism. This makes it difficult to optimize response strategies based on user feedback, resulting in generated responses that fail to fully meet the user's emotional needs. The second category uses neural networks to directly generate dialogue content, but often suffers from inaccuracies and ambiguities in emotional expression, especially when the dataset is insufficient or the emotions are complex, resulting in responses that fail to achieve the expected emotional support. The third category uses knowledge-enhanced generative techniques, enriching the generated dialogue content by introducing knowledge graphs, parameterized knowledge, or unstructured text. This type of method improves the richness and consistency of the dialogue to some extent, but still faces the problem of generated content not matching reality and relies heavily on the quality of external knowledge, easily leading to insufficient emotional expression. Therefore, how to accurately predict appropriate response strategies and generate responses consistent with those strategies is an extremely important technical problem to be solved. Summary of the Invention

[0004] To address the issues of existing technologies struggling to accurately predict and generate responses consistent with the chosen strategy, this invention proposes an emotion-support dialogue strategy management method that integrates user feedback. This method effectively and accurately predicts and generates responses consistent with the chosen strategy.

[0005] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows:

[0006] A method for managing emotional support dialogue strategies that incorporate user feedback, comprising the following steps:

[0007] S1. Obtain the input dialogue information;

[0008] S2. Input the dialogue information into a pre-trained knowledge augmentation encoder and output a contextual representation of the augmented knowledge;

[0009] S3. Based on the context representation, select a strategy to generate a response that reassures the user;

[0010] S4. Based on the response strategy and the context representation, generate a candidate response consistent with the response strategy;

[0011] S5. Predict the user's feedback to the candidate responses, and iteratively generate the optimal response strategy based on the feedback.

[0012] Preferably, constructing the knowledge-enhanced encoder includes: using an adapter to inject psychological knowledge into a preset language model PLM to obtain the knowledge-enhanced encoder.

[0013] Preferably, the step of selecting a response strategy to appease the user based on the context representation includes:

[0014] Construct a high-level state graph representing the relationship between the context representation and the response strategy. The high-level state graph contains nodes V and edges E, where node V represents a set of utterance representations. The strategy set is represented as s, and the edge E includes the connection between two consecutive utterances, as well as the connection between the supporter's utterance and its specific strategy;

[0015] Several stacked graph convolutional network layers are used to model the correlation between statements and policies involved in the high-level state graph, resulting in the graph representation G of the high-level state graph. * ;

[0016] The representation of the high-level state graph is captured using a neural network based on a self-attention mechanism, resulting in the following expression used to assist in the high-level state representation:

[0017]

[0018] in, The graphical representation at timestamp t. Let t be the high-level state representation, which is used to assist the policy selection network and policy evaluation network in generating response policies.

[0019] Preferably, the policy selection network generates the following expression for the response policy based on the high-level state representation:

[0020]

[0021] in, This represents the high-level action, i.e., the response strategy, and tanh represents the first activation function. This represents the first weight vector. This represents the first bias parameter.

[0022] Preferably, the strategy evaluates the network input high-layer state. and high-level actions The expression for predicting the current Q value is as follows:

[0023]

[0024] Where ReLU represents the second activation function. This represents the second weight vector. Represents the third weight vector. This represents the second bias parameter.

[0025] Preferably, the step of generating a candidate response consistent with the response strategy based on the response strategy and the context representation is as follows:

[0026]

[0027] in, This represents the underlying action of expressing responses, namely the generation of candidate responses; Decoder represents the decoder. This represents a contextual representation of knowledge based on psychology. This indicates that the response strategy is embedded.

[0028] Preferably, S5 specifically includes: using a preset reward model to predict the user's feedback to the candidate response; if the feedback is positive, then the candidate response is used as the final output of the optimal response strategy; if the feedback is negative, then a hierarchical reinforcement learning method is used to iteratively generate the optimal response strategy again.

[0029] Preferably, the loss function used to train the reward model is expressed as follows:

[0030]

[0031] Where loss(.) represents the loss function, r θ (q,p i The reward model targets mental health issues q and responses p. i The scalar output, θ represents the model parameters, p i ∈(p0,p1), i represents the response p i The number of data points, where D represents the dataset with human rankings and σ represents the standard deviation.

[0032] This invention proposes an emotion-support dialogue strategy management system that integrates user feedback, comprising:

[0033] The input module is used to input dialogue information;

[0034] A knowledge-enhancing encoder is used to receive the dialogue information and output a contextual representation of the enhanced knowledge;

[0035] The response strategy selection unit is used to select a response strategy to reassure the user based on the context representation.

[0036] A strategy consistency generation unit is used to generate a candidate response consistent with the response strategy based on the response strategy and the context representation;

[0037] The user feedback prediction unit is used to predict user feedback to the candidate responses and iteratively generate the optimal response strategy based on the feedback.

[0038] Preferably, the step of generating the optimal response strategy iteratively based on the feedback includes: using a preset reward model to predict the user's feedback to the candidate response; if the feedback is positive, then the candidate response is used as the final output of the optimal response strategy; if the feedback is negative, then a hierarchical reinforcement learning method is used to iteratively generate the optimal response strategy again.

[0039] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0040] This invention proposes an emotion support dialogue strategy management method that integrates user feedback. First, a knowledge-enhanced encoder is used to encode the input dialogue information to better encode the context and accurately perceive the user's psychological state. Second, a response strategy is selected based on the context representation to identify responses that can soothe the user. Then, based on the context representation and the response strategy, relevant semantic and linguistic details are captured to generate candidate responses consistent with the chosen strategy. Finally, the user's feedback to the candidate responses is predicted, and an optimal response strategy is generated iteratively. This allows for learning the cognitive-level strategy selection and the expression-level strategy consistency response generation process, thereby effectively and accurately predicting suitable response strategies and generating responses consistent with those strategies. Attached Figure Description

[0041] Figure 1 Example diagram illustrating the value of existing user feedback;

[0042] Figure 2 Example diagram illustrating the value of existing psychological knowledge;

[0043] Figure 3This is a flowchart illustrating an emotion support dialogue strategy management method that integrates user feedback, as proposed in an embodiment of the present invention.

[0044] Figure 4 This is a flowchart illustrating the principle of an emotion support dialogue strategy management method that integrates user feedback, as proposed in an embodiment of the present invention.

[0045] Figure 5 This diagram illustrates the structural block of an emotion support dialogue strategy management system that integrates user feedback, as proposed in this embodiment of the invention. Detailed Implementation

[0046] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this embodiment.

[0047] It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings;

[0048] To facilitate understanding of this embodiment, the prior art information of this embodiment is first introduced as follows:

[0049] Due to insufficient understanding of response strategies, existing methods often suffer from a mismatch between predicted and actual response strategies. For example, the system might predict a "questioning" strategy, such as asking "Why are you sad?", but the generated response might be inadequate, such as "Sadness is a common emotion." Such "information-based" responses are unlikely to alleviate the user's emotions.

[0050] User feedback plays a crucial role in emotional support dialogue systems. It not only measures user satisfaction with generated responses but also directly influences the direction and content of the conversation. For example... Figure 1 As shown, feedback can help machines more accurately understand the issues causing negative emotions in users, thereby optimizing their responses. For example, when a user faces unemployment, a lack of feedback might lead the machine to provide statements that don't align with the user's actual situation, failing to alleviate their confusion and anxiety. By considering user feedback, the machine can better understand the specific difficulties the user faces by guiding them to share those difficulties. Based on this feedback, the dialogue system can more accurately grasp the user's needs and provide more precise and practical personalized suggestions, such as recommending jobs in specific industries or offering targeted training programs. This makes users feel understood and gives them more confidence to cope with difficulties, calming their emotions. However, accurately perceiving and understanding user feedback is not easy. Furthermore, using feedback to adjust dialogue strategies and solve problems is also very challenging.

[0051] Furthermore, mental health knowledge plays a crucial role in emotional support dialogue systems. It helps the system more accurately understand users' emotional states and psychological needs, thereby providing more targeted advice and solutions. However, current research has neglected the incorporation of mental health knowledge, resulting in a lack of professionalism in the generated content. For example... Figure 2 As shown, machines rely solely on simple matching to retrieve relevant text, lacking specificity and resulting in vague suggestions that fail to alleviate users' confusion and uncertainty. Furthermore, the retrieval process may introduce outdated or irrelevant information, further reducing the effectiveness of the suggestions. By incorporating parameterized mental health knowledge, machines can gain a deeper understanding of users' emotional states and provide specific and personalized advice. Therefore, the introduction of psychological knowledge can effectively compensate for the limitations of text-based retrieval, improve the quality of responses, and enhance users' trust and satisfaction with the results.

[0052] To address the aforementioned issues, this embodiment proposes a dialogue strategy management framework that integrates user feedback. This framework can accurately perceive the user's emotional state based on context and psychological knowledge, and then select an appropriate strategy to generate a suitable response.

[0053] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] Example 1

[0055] This embodiment proposes a method for managing emotion-supportive dialogue strategies that integrates user feedback. Emotion-supportive dialogue aims to generate supportive responses through carefully designed strategies, thereby helping the dialogue system alleviate users' negative emotions. The input is set as context C = [u1,…,u…]. t The output is a supportive response y with a specific strategy. This embodiment uses a hierarchical reinforcement learning method, treating the emotion support dialogue as a two-level task including a strategy level and an expression level. The user is regarded as the environment, and the model is responsible for completing the two-level tasks. First, the high-level (i.e., the strategy level) task is based on the context C=[u1,…,u t Choose response strategy a h When selecting response strategy a h After observing the context, the low-level (i.e., expression layer) task is to generate candidate responses consistent with the policy. l Subsequently, candidate responses are evaluated based on user feedback and treated as reward signals from the environment, which are then used to train the reward model. The concepts involved are as follows:

[0056] High-level state s h :s h It includes the history of the dialogue and the set of strategies used in the discourse of supporters.

[0057] High-level action a h High-level actions refer to choosing a response strategy. h It is considered the selected strategy.

[0058] High-level rewards r h High-level rewards are scalars used to evaluate the selected strategy, measured by the accuracy of the selected strategy and the feedback generated by the corresponding response.

[0059] Low-level state s l :s l It includes the dialogue history, and each supporter's statement is not labeled with a strategy tag, but also includes strategy constraints.

[0060] Low-level action a l Low-level actions refer to generating responses. l It is considered as the generated response.

[0061] Low-level reward r l Low-level rewards are scalars used to evaluate user feedback.

[0062] like Figure 3 and Figure 4 As shown, the method includes the following steps:

[0063] S1. Obtain the input dialogue information;

[0064] S2. Input the dialogue information into a pre-trained knowledge augmentation encoder and output a contextual representation of the augmented knowledge;

[0065] In S2, specialized psychological knowledge is introduced to enhance the contextual representation, thereby enabling a better understanding of user problems, providing reassurance, and offering more reliable solutions. However, when integrating psychological knowledge, relying on corpus-based retrieval of similar text may introduce noise, leading to irrelevant or unsuitable content, thus impacting performance. Furthermore, injecting new knowledge or knowledge bases into the machine through fine-tuning requires large-scale retraining. Therefore, constructing the knowledge-enhanced encoder involves: using an adapter to inject psychological knowledge into a pre-defined language model PLM, resulting in the knowledge-enhanced encoder; avoiding large-scale training. An adapter is a lightweight model component that combines a pre-trained general model with domain-specific knowledge, thereby improving the model's performance in that domain. This step uses a plug-and-play paradigm for knowledge injection, injecting the knowledge base into the frozen language model through a knowledge plugin, i.e., an adapter. Specifically, a knowledge adapter KA is trained, and the pre-trained language model PLM is enhanced with psychological knowledge. Given a pre-trained language model PLM as the backbone, psychological knowledge K is injected into P by training KA and freezing the parameters of PLM. Here, K is a question-and-answer corpus on mental health issues called MentalQA. The KA (Knowledge Adapter) is trained using a masked language model task. To ensure the adapter learns not only language skills but also psychological knowledge, a random masking method using non-stop words is employed. After training, the KA can inject psychological knowledge into the language model in a plug-and-play manner. The knowledge adapter is then used to enrich the contextual representation for this psychological knowledge injection. For example, given u... t This indicates the utterance at timestamp t, and the result is obtained. Indicates speaking u t The hidden vector, Indicates speaking u t The `Encoder` represents the utterance, and the `Programmer` represents the encoder. This combines the generality of a pre-trained language model with the domain expertise of an adapter, effectively improving the performance of the dialogue system in the mental health domain without significantly increasing additional parameters or complexity. Furthermore, the adapter, optimized for a specific domain, better adapts to the context and needs of the mental health domain, providing more accurate and professional responses. By leveraging the generality of the pre-trained language model and the adapter to model specialized knowledge in the mental health domain, the performance of the dialogue system in the mental health domain can be effectively improved without adding too many additional parameters or complexity. Simultaneously, because the adapter is a domain-optimized model, it better adapts to the context and needs of the mental health domain, providing more effective responses.

[0066] S3. Based on the context representation, select a strategy to generate a response that reassures the user;

[0067] In S3, based on the context representation, a response strategy to appease the user is selected. To better learn the relationship between the response strategy and the context, a graph-based method is designed, including:

[0068] A high-level state graph is constructed to represent the relationship between the context representation and the response strategy. The high-level state graph contains nodes V and edges E. User utterances, supporter utterances, and strategies in supporter utterances are all represented as nodes. Therefore, node V represents a set of utterance representations. And the policy set is represented as s, for example, Define a graph containing n nodes, denoted as V = v i ,1≤i≤n. Edge E includes the connection between two consecutive utterances, and the connection between the supporter's utterance and its specific strategy; (1) the connection between two consecutive utterances, namely This connection maintains the coherence and sequential relationship of the dialogue. (2) The connection between the supporter's statements and their specific strategies, i.e., h u →s. This connection identifies the strategy adopted by the supporter in their response. By constructing a state graph of this strategy layer, it is easy to grasp the relationship between the response strategy and the context. The graph-building approach is used to learn the states of the strategy layer, with the aim of capturing patterns in strategy selection.

[0069] Using L G A stacked graph convolutional network (GCN) layer is used to model the correlation between statements and policies involved in the high-level state graph, resulting in a graph representation G of the high-level state graph. * The aim is to obtain the policy layer state representation. Since the high-level state graph is initialized by the knowledge adapter, the representation of node v in the (g+1)th layer is updated according to the proposed method:

[0070]

[0071] in, Let v be the set of direct neighbors of node v, and Λ be the normalized matrix. These are two learnable weight matrices of the g-th layer. After L... G After stacking several graph convolutional network (GCN) layers, a graph representation G is obtained. * .

[0072] The representation of the high-level state graph is captured using a Transformer neural network based on a self-attention mechanism. By processing the input sequence through a Transformer layer, the expression used to assist in the high-level state representation is obtained as follows:

[0073]

[0074] in, The graphical representation at timestamp t. Let t be the high-level state representation, which is used to assist the policy selection network and policy evaluation network in generating response policies.

[0075] The strategy selection network generates a response strategy based on the high-level state representation. Specifically, it will... The expression input to the fully connected layer to generate the t-th higher-level action is as follows:

[0076]

[0077] in, This represents the high-level action, i.e., the response strategy, and tanh represents the first activation function. This represents the first weight vector. Indicates the first bias parameter;

[0078] The goal of a policy evaluation network is to calculate the Q-score, which represents the total reward after selecting a particular policy. The higher-level Q-score also determines whether the policy selected by the policy selector matches the current higher-level state, and is formalized as follows:

[0079]

[0080] in This refers to the high-level reward after strategy selection. The reward includes two important aspects: the consistency between the expected response strategy and the actual strategy adopted, and the user's feedback on the response generated by the selected strategy. γ∈[0,1] is a discount factor that takes into account the impact of location bias in actual application, ensuring that the strategy can obtain a higher reward. In this case, the strategy evaluator can capture the user's preference for the strategy level at each location.

[0081] Specifically, a fully connected layer is used to estimate the Q-value. At position t, the policy evaluates the network input high-level state. and high-level actions The expression for predicting the current Q value is as follows:

[0082]

[0083] Where ReLU represents the second activation function. This represents the second weight vector. Represents the third weight vector. This represents the second bias parameter. Based on the expected Q value, the policy selector can update its corresponding parameters, thereby improving performance.

[0084] S4. Based on the response strategy and the context representation, generate a candidate response consistent with the response strategy;

[0085] In S4,

[0086] To achieve policy-consistent responses, this unit aims to leverage psychologically enhanced contextual representations and policy constraints. Generate candidate responses. We first infer the policy vector, and then generate responses consistent with the selected policy.

[0087] To avoid inconsistencies between policy selection and response content (i.e., inaccurate wording), a latent diffusion model is employed to learn policy concepts. The goal is to encode policy concepts as intermediate representations in a pre-trained model and leverage semantic priors to guide the linguistic expression of these concepts. To capture unique expressions and linguistic structures related to the policy, policy vectors are introduced into the embedding space of the pre-trained language model. Specifically, a novel pseudoword 'a' is used... * This is used to represent the strategy to be learned. The pseudo-word is treated as a regular word and used to generate responses consistent with the corresponding strategy. During optimization, a set of examples demonstrating this dialogue strategy in different scenarios is input, and neutral text from the template is randomly sampled as conditional input, similar to "a". * Example" or "a * The description is as follows. The optimization objectives are as follows:

[0088]

[0089] Among them, c θ (p) represents the model that maps the conditional input p to a conditional vector, t represents the time step, and z represents the time vector. t Let represent the potential noise at time t, and ∈ be an unscaled noise sample. θ This is a denoising network. The goal of this task is to correctly remove noise added to the latent representation of the policy response. It is expected that the embedding vectors learned through this process will capture the unique expression and linguistic structure of the policy. This is achieved by providing the diffusion model with a response policy that conforms to the specific policy. An example was obtained, showing the embedded response strategy. This is used to generate customized responses that conform to the response strategy. (The response strategy embedding is then obtained.) Then, consider using knowledge-rich contextual representations. (Where C represents the context) Generate candidate responses. To generate responses consistent with the selected response strategy, the response strategy is embedded. Treated as regular word embeddings, the following candidate responses, consistent with the aforementioned response strategy, are generated:

[0090]

[0091] in, This represents the underlying action of expressing responses, namely the generation of candidate responses; Decoder represents the decoder. This represents a contextual representation of knowledge based on psychology. This indicates the embedding of the response strategy. By treating the response strategy as a new pseudo-word in the model's vocabulary, it can be used to guide the language generation process.

[0092] S5. Predict the user's feedback to the candidate responses, and iteratively generate the optimal response strategy based on the feedback.

[0093] In this embodiment, an encoder incorporating psychological knowledge is first constructed to perceive the user's psychological state. Through a knowledge adapter, this encoder can parameterize this psychological knowledge, reducing noise interference during the retrieval process. Considering that graph structures are better at capturing the relationship between context and response strategies, this embodiment constructs a graph-based strategy selection unit to select the best appeasing strategy from the strategy pool based on the perceived user state. Subsequently, this embodiment employs a latent diffusion model to learn the strategy embedding, capturing the corresponding semantic and linguistic details to generate a response consistent with the strategy. Finally, this embodiment designs a training model using a hierarchical reinforcement learning method. This method uses a reward model to evaluate the user's feedback to potential responses, iteratively finding the optimal strategy and corresponding response. Experimental results on benchmark datasets demonstrate the superior performance of this method. The method proposed in this embodiment has significant practical applications, especially in areas such as mental health care, emotional support dialogue, and psychological services for the elderly. By introducing a user feedback-driven parametric knowledge enhancement framework, this method can perceive the user's emotional state in real time and dynamically adjust the response strategy, thereby providing more personalized, targeted, and effective emotional support. This technology, based on the infusion of psychological knowledge and the generation of strategy consistency, can enhance the machine's ability to understand and respond to user needs in emotionally complex and ever-changing dialogue scenarios. Therefore, this method can effectively improve the intelligence and practicality of dialogue systems, promoting the application and development of intelligent dialogue technology in mental health services.

[0094] It is also important to note that traditional methods for generating responses in emotional support dialogue tasks typically retrieve text fragments from a corpus that match the context. However, these methods ignore user feedback and cannot dynamically adjust response strategies based on the user's emotional state. Existing methods are suitable for generating relatively fixed responses but struggle to effectively alleviate negative user emotions. Furthermore, these methods are prone to introducing noise during the retrieval process, resulting in responses that do not accurately match user needs. To address these shortcomings, this embodiment proposes a novel framework that overcomes the limitations of traditional emotional support dialogue systems, effectively improving the personalization of dialogue and user satisfaction. Specifically, this embodiment constructs a hierarchical reinforcement learning model based on user feedback by injecting professional knowledge from the field of mental health into a pre-trained language model in the form of an adapter. While traditional methods have been applied in emotion generation tasks, they often cannot handle complex psychological issues. This embodiment, by combining mental health knowledge with user feedback, can dynamically adjust response strategies and generate responses consistent with the strategies. In addition, this embodiment uses a latent diffusion model to learn strategy concepts, ensuring that the content of the generated responses conforms to the established dialogue strategy. The advantages of the method proposed in this embodiment include:

[0095] We propose an emotion dialogue framework based on hierarchical reinforcement learning, which can adjust strategies based on user feedback to generate personalized, reassuring responses and improve user experience.

[0096] By using an adapter to inject mental health knowledge into a pre-trained language model, it is possible to better understand the user's mental state and generate personalized suggestions based on the user's specific situation, significantly reducing the noise problem in traditional methods.

[0097] By learning policy embeddings through a latent diffusion model, the generated response content is ensured to be consistent with the selected policy. Furthermore, a feedback estimation unit is used to ensure that the generated response meets the user's expectations both logically and in terms of common sense.

[0098] Furthermore, to further evaluate the performance of the emotion-supported dialogue strategy management method integrating user feedback proposed in this embodiment, the ESConv dataset was used in the experiments. This dataset is widely used in emotion-supported dialogue tasks. For ease of analysis, this embodiment employed both automated and manual evaluation metrics. The automated evaluation metrics used in the experiments included PPL, BLEU, DIST, ROUGE-L, Meteor, and CIDEr, which are widely used in dialogue generation tasks. Among them, PPL (Perplexity) measures the effectiveness of the language model; the lower the score, the better the model performance. BLEU analyzes the co-occurrence of generated responses and real text in n-tuples. DIST measures the diversity of the corpus by calculating the ratio of all unique n-tuples in the output. ROUGE-L (Longest Common Subsequence) evaluates the accuracy of the generated text based on the longest common subsequence between the candidate sentence and the target sentence. Meteor is used to evaluate the precision and recall of the generated text. CIDEr measures the response quality by calculating the similarity of n-tuples between the generated text and the reference text. The human evaluation component involves recruiting participants to subjectively assess the quality of the generated responses. The evaluation dimensions used include fluency (Flu), recognizability (Ide), comfort (Com), suggestion (Sug), and overall performance (Ove). Participants are required to compare the response performance of the two different models across these dimensions based on the generation of each round of dialogue. Experimental results show that this method significantly outperforms traditional methods.

[0099] Example 2

[0100] This embodiment further explains S5. After generating candidate responses with consistent response strategies in the modeling expression layer, the user's feedback to the candidate responses is evaluated. A preset reward model is used to predict the user's feedback to the candidate responses to determine whether the response effectively interacts with the user, and this is then used as the final output. If the feedback is positive, the candidate response is used as the final output of the optimal response strategy. If the feedback is negative, a hierarchical reinforcement learning method is used to iteratively generate the optimal response strategy again.

[0101] However, the benchmark dataset lacks user feedback due to incomplete information. Therefore, this paper proposes learning user feedback by fine-tuning a reward model. This reward model takes context and the response as input and outputs a rating for that response. To train the reward model, a pre-trained language model and randomly initialized linear layers are used. Drawing inspiration from Stiennon's paper "Learning to summarize with human feedback," this reward model is based on a mental health knowledge base, MentalQA, which contains rankings of multiple responses from users to the same mental health question. Specifically, the rankings are obtained through user "likes," avoiding personal bias. Given a mental health question q, the reward model predicts which response p∈(p0,p1) is better. i When favored, the loss function used to train the reward model is expressed as follows:

[0102]

[0103] Where loss(.) represents the loss function, r θ (q,p i The reward model targets mental health issues q and responses p. i The scalar output, θ represents the model parameters, p i ∈(p0,p1), i represents the response p i The number of responses is denoted by D, where D represents the dataset with human rankings, and σ represents the standard deviation. At the end of training, the output of the reward model is normalized so that the average score of all responses in the dataset is 0.

[0104] The process of iteratively generating the optimal response strategy using a hierarchical reinforcement learning method is as follows:

[0105] To ensure rapid and stable convergence of hierarchical reinforcement learning, a multifaceted objective was designed to jointly model the loss functions of both levels. In each dialogue round, a state graph is first constructed, and a response strategy is selected based on the graph representation.

[0106] Specifically, the policy executor is responsible for selecting the response policy, while the policy evaluator calculates the Q-score. They are trained using the classic mean squared error loss function:

[0107]

[0108] θ is the target Q-value at time t, which consists of the current reward and future Q-values ​​from the target network. h and θ h′ It is the parameter set of the strategy evaluator, the difference being θ hUpdated during training, while θ h′ It is a set of previously experienced parameters that are fixed during optimization. This indicates the target strategy of the strategy executor.

[0109] The Q-value generated by the policy evaluator is used to update the policy executor through policy gradients. Actions with higher Q-values ​​are generated by maximizing the overall ranked expected return.

[0110]

[0111] Where a h =μ h (s h |φ h The ) represents the response policy chosen by the policy executor through its policy selection. Ultimately, the losses of the lower-level agent are aggregated as follows:

[0112] L h =L(θ) h )+L(φ h ).

[0113] Next, psychological knowledge is integrated into the low-level state representation using a knowledge adapter. This adapter is trained using a masked language model (MLM) loss. Then, based on the selected policy and the knowledge-enhanced representation, a policy-consistent response is generated using response policy embeddings.

[0114]

[0115] It is the low-level target Q value at time point t. This represents the low-level reward, which consists of the current reward and the similarity between the generated response and the reference response. θ l and θ l′ This is the parameter set for lower-level evaluators. θ l Updated during training, while θ l′ This is the set of previously experienced parameters fixed during optimization. The loss generated by low-level consistent generation is:

[0116]

[0117] Where μ l Refers to the low-level response generation model, a l =μ l (s l |φ l This indicates that the lower-level action performer generates a policy-consistent response through the language model.

[0118] Then the loss L(θ) generated by the response l ) and the loss L(φ) in evaluating the response feedback l )Sum:

[0119] L l =L(θ) l )+L(φ l ).

[0120] Finally, candidate responses are evaluated by estimating user feedback, and strategies and responses are adjusted based on rewards. The estimated loss is as described in the formula above.

[0121] Example 3

[0122] See Figure 5 This embodiment proposes an emotion support dialogue strategy management system that integrates user feedback, including:

[0123] The input module is used to input dialogue information;

[0124] A knowledge-enhancing encoder is used to receive the dialogue information and output a contextual representation of the enhanced knowledge;

[0125] The response strategy selection unit is used to select a response strategy to reassure the user based on the context representation.

[0126] A strategy consistency generation unit is used to generate a candidate response consistent with the response strategy based on the response strategy and the context representation;

[0127] The user feedback prediction unit is used to predict user feedback to the candidate responses and iteratively generate the optimal response strategy based on the feedback.

[0128] The step of generating the optimal response strategy based on the feedback iteratively includes: using a preset reward model to predict the user's feedback to the candidate response; if the feedback is positive, then the candidate response is output as the optimal response strategy; if the feedback is negative, then a hierarchical reinforcement learning method is used to iteratively generate the optimal response strategy again.

[0129] In this embodiment, a knowledge-enhanced encoder is first used to encode the input dialogue information to better encode the context and accurately perceive the user's psychological state. Secondly, a response strategy is selected based on the context representation to identify a strategy that can soothe the user. Then, based on the context representation and the response strategy, corresponding semantic and linguistic details are captured to generate candidate responses consistent with the chosen strategy. Finally, the user's feedback to the candidate responses is predicted, and an optimal response strategy is generated iteratively. This facilitates learning the cognitive-level strategy selection and the expression-level strategy consistency response generation process, thereby effectively and accurately predicting the appropriate response strategy and generating responses consistent with it.

[0130] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for managing emotional support dialogue strategies that integrates user feedback, characterized in that, Includes the following steps: S1. Obtain the input dialogue information; S2. Input the dialogue information into a pre-trained knowledge augmentation encoder and output a contextual representation of the augmented knowledge; S3. Based on the aforementioned context representation, select a strategy for generating a response to appease the user, including: Construct a high-level state graph representing the relationship between the context representation and the response strategy, the high-level state graph containing nodes. and edge ,node Represented as a set of discourses and strategy set representation ,side This includes the connection between two consecutive discourses, as well as the connection between the supporter's discourse and its specific strategy; Several stacked graph convolutional network layers are used to model the correlation between statements and policies involved in the high-level state graph, resulting in a graph representation of the high-level state graph. ; The representation of the high-level state graph is captured using a neural network based on a self-attention mechanism, resulting in the following expression used to assist in the high-level state representation: in, For timestamps The time diagram shows, For the first A high-level state representation is used to assist the policy selection network and the policy evaluation network in generating response policies; The strategy selection network generates the following expression for the response strategy based on the high-level state representation: in, This indicates a high-level action, i.e., the aforementioned response strategy. This represents the first activation function. This represents the first weight vector. Indicates the first bias parameter; The strategy evaluates the network input high-level state. and high-level actions The expression for predicting the current Q value is as follows: in, This represents the second activation function. This represents the second weight vector. Represents the third weight vector. Indicates the second bias parameter; S4. Based on the response strategy and the context representation, generate a candidate response consistent with the response strategy; S5. Predict the user's feedback to the candidate responses, and iteratively generate the optimal response strategy based on the feedback.

2. The method for managing emotion-supported dialogue strategies that integrate user feedback according to claim 1, characterized in that, The construction of the knowledge-enhancing encoder includes: using an adapter to inject psychological knowledge into a preset language model PLM to obtain the knowledge-enhancing encoder.

3. The emotional support dialogue strategy management method integrating user feedback according to claim 2, characterized in that, Based on the response strategy and the context representation, the following candidate responses consistent with the response strategy are generated: in, This indicates the underlying action, namely the generation of candidate responses. Indicates decoder, This represents a contextual representation of knowledge based on psychology. This indicates that the response strategy is embedded.

4. The emotional support dialogue strategy management method integrating user feedback according to claim 3, characterized in that, S5 specifically includes: using a preset reward model to predict the user's feedback to the candidate response; if the feedback is positive, the candidate response is used as the final output of the optimal response strategy; if the feedback is negative, a hierarchical reinforcement learning method is used to iteratively generate the optimal response strategy again.

5. The emotional support dialogue strategy management method integrating user feedback according to claim 4, characterized in that, The expression for the loss function used to train the reward model is as follows: in, Represents the loss function. This indicates that the reward model addresses mental health issues. and reply scalar output, Indicates model parameters, , Indicates reply Quantity, This represents a dataset with human rankings. It represents the standard deviation.

6. A system for managing emotional support dialogue strategies that integrates user feedback, the system being implemented based on any one of the emotional support dialogue strategy management methods that integrate user feedback as claimed in claims 1-5, characterized in that, include: The input module is used to input dialogue information; A knowledge-enhancing encoder is used to receive the dialogue information and output a contextual representation of the enhanced knowledge; The response strategy selection unit is used to select a response strategy to reassure the user based on the context representation. A strategy consistency generation unit is used to generate a candidate response consistent with the response strategy based on the response strategy and the context representation; The user feedback prediction unit is used to predict user feedback to the candidate responses and iteratively generate the optimal response strategy based on the feedback.

7. The emotion support dialogue strategy management system integrating user feedback according to claim 6, wherein the step of iteratively generating the optimal response strategy based on the feedback includes: The user's response to the candidate response is predicted using a preset reward model. If the response is positive, the candidate response is output as the optimal response strategy. If the feedback is negative, a hierarchical reinforcement learning method is used to iteratively generate the optimal response strategy.

Images