Emotion support dialogue system based on variational bayesian inverse reinforcement learning strategy

Abstract

The application discloses a kind of based on variational bayes inverse reinforcement learning strategy emotional support dialogue system, its characteristics are to adopt including: emotional support strategy selection module, dialogue common sense analysis reasoning module and reply generation module are constructed strategy emotional support dialogue system, the emotional support strategy selection module is by dialogue state encoder and variational bayes inverse reinforcement learning strategy selection module, according to current dialogue history information and the strategy used to carry out strategy selection;The dialogue common sense analysis reasoning module generates and predicts the common sense information analysis relevant to current dialogue;The reply generation module will generate content and dialogue history generation reply.The application compared with prior art has enhanced the information richness and accuracy of generated reply, effectively improves the strategy selection accuracy and user's experience in the strategy emotional support dialogue scene, with good use prospect and commercial development value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of policy emotional support dialogue system, in particular to an emotional support dialogue system based on variational bayes inverse reinforcement learning policy. BACKGROUND

[0002] Emotional support dialogue systems have recently attracted attention due to the demand for mental health in people's daily life. Such systems gradually understand the user's plight in a guided manner by interacting with the user in a dialogue, provide appropriate psychological comfort to the user, and ultimately provide appropriate solutions and suggestions to the user. Daily emotional support and psychological comfort is an important aspect of maintaining mental health, which involves an individual's attitude and stress towards life, work, social relationships, etc. Currently, friends or professionals cannot understand the emotional state of individuals in real time because of limited communication time and lack of professional psychological knowledge, resulting in relatively low communication efficiency.

[0003] In order to provide better emotional support, an emotional support dialogue dataset ESConv annotated with 8 support strategies has been proposed in recent years. In this task, the dialogue system will select the appropriate emotional support strategy according to the current dialogue history. The basic framework of the emotional support strategy is a three-stage model of "exploration-understanding-action", so selecting the appropriate dialogue strategy will more reasonably guide the generation of dialogue replies.

[0004] The prior art reinforcement learning performs policy selection, and the reward function is artificially set to be the change in the future mood of the seeker after adopting the policy of the current model. In this process, single-step user simulation, i.e. environment simulation, is involved. However, in the actual emotional support task, the emotional support dialogue dataset has high professionalism, and its environment and task goals are complex and difficult to simulate. In the process of exploring the environment, there is a lack of expert strategy annotation, and the design of the reward function also needs to consider the complex potential influencing factors in the dialogue scenario. SUMMARY

[0005] The emotional support dialogue system based on the variational Bayesian inverse reinforcement learning strategy aims at the deficiencies of the prior art, adopts the emotional support dialogue system constructed by an emotional support strategy selection module, a dialogue common sense analysis reasoning module and a reply generation module, the emotional support strategy selection module selects a strategy according to current dialogue history information and a strategy that has been used, the information richness of the reply generated by the dialogue system is improved through support strategy selection, dialogue common sense generation and analysis, the accuracy of strategy selection is greatly improved, the method effectively improves the strategy selection accuracy in the strategy emotional support dialogue scene and the experience of the user, only needs to learn from the existing data set, learns the preferences of the expert track, does not perform artificial setting of a reward function, can enhance the information richness and accuracy of the generated reply, and improves the accuracy of strategy selection, has a good application prospect and commercial development value.

[0006] The specific technical scheme for achieving the object of the present application is as follows: an emotional support dialogue system based on a variational Bayesian inverse reinforcement learning strategy, characterized by adopting an emotional support dialogue system constructed by an emotional support strategy selection module, a dialogue common sense analysis reasoning module and a reply generation module, generating a reply of the dialogue system through support strategy selection, dialogue common sense generation and analysis, the emotional support strategy selection module is composed of a dialogue state encoder and a variational Bayesian inverse reinforcement learning strategy selection module, the module selects an emotional support strategy according to current dialogue history information and a strategy that has been used, and specifically includes:

[0007] 1-1: the dialogue state encoder converts the dialogue history C and the list of previously used strategies S into a feature matrix, and the processing process is represented by (a)-(f) as follows:

[0008] C=(y0,x0,y,x,…,y t ) (a);

[0009] H t =( [CLS ],y0,[SEP ],x0,[SEP ],y,[SEP ],x,[SEP ],…,y t ) (b);

[0010] S t =(s0,s,s2,…,s t- ) (c);

[0011] E H =Roberta(H t ) [Pooler_Output] (d);

[0012] E HL =Linear(E H ) (e);

[0013] E H+t =[E HL ;S t ] (f);

[0014] where t represents the round of the current dialogue; x represents the content of the model previously in the reply; y represents the historical statement of the user; H t represents processing the dialogue history into a suitable format, inputting it into the Roberta-base model for encoding, and obtaining the pooling layer output vector E H of the model, then inputting it into the linear layer for dimension conversion, obtaining E HL , and finally splicing with the strategy list S t already used in the previous historical dialogue, obtaining the comprehensive state encoding E HSt of the current dialogue history.

[0015] 1-2: The variational Bayesian inverse reinforcement learning strategy selection module predicts the strategy according to the current dialogue history, and the variational Bayesian inverse reinforcement learning method is used to train the strategy selection module in this module, which mainly includes a reward function fitting module and a strategy selection module. The implementation steps of the variational Bayesian inverse reinforcement learning method are as follows: first, the original data set can be encoded by the dialogue state encoder mentioned above, and the specific construction method is as follows: for each complete dialogue in the data set, the dialogue roles are divided into the seeker and the helper, the dialogue history before the helper's reply is state encoded by the dialogue state encoder, and then the reply strategy used by the helper is added, so as to obtain the sequence data D raw , m represents the number of complete dialogues in the data set, and τ represents the maximum number of rounds of the current dialogue. Then the data in D raw is further divided, and the obtained training data is a four-tuple, which includes the previous dialogue history state encoding, the strategy used by the helper in the previous state, the subsequent dialogue history state encoding, and the strategy used by the helper, n represents the number of four-tuples finally obtained.

[0016] The sequence data D raw is represented by the following formula (g):

[0017]

[0018] where, represents the comprehensive state encoding of the tth round of dialogue history of the ith dialogue in the data set, represents the strategy selection of the tth round of dialogue in the ith dialogue, t represents the current round of dialogue, and τ Im represents the number of complete dialogues contained in the current data set.

[0019] D raw The training data D obtained by further segmentation is represented by the following formula (h):

[0020]

[0021] wherein, represents the total number of training data, represents the comprehensive state encoding of the dialogue history segmented from D raw i represents the use strategy segmented from D raw represents the state encoding of the dialogue in the original data set represents the state encoding of the dialogue in the original data set i represents corresponding strategy action selection.

[0022] The reward function fitting module includes a deep learning neural network RewardEncoder to learn the reward function variational posterior distribution, and the reward function is set as a Gaussian distribution, wherein the composition of RewardEncoder is as follows, the input value is the comprehensive state encoding of the dialogue history, W1, W2, b1 and b2 are trainable parameters, h1 represents an intermediate result vector, and finally the module outputs a vector with a dimension of 2, wherein the first dimension represents the mean of the current reward function distribution mean, and the second dimension represents the logarithmic standard deviation log_sds, and the intermediate result vector h1 is represented by the following formula (i):

[0023] h1=ReLU(W1E HSt +b1)

[0024] (mean,log_sds)=W2h1+b2 (i);

[0025] The policy selection module adopts the design method of Dueling Q-Network, wherein the input value is the comprehensive state encoding of the dialogue history, W3, W4, W V , W A , b3, b4, b V and b A are trainable parameters, and a represents the action of the policy selected by the policy selection model, represents the dimension of the action space, represents the global value of the current state, represents the advantage value of each action under a given state, x1 and x2 are respectively the activation values of two hidden layers; is the state​​ Q value when action a is taken;

[0026] The activation value x1 of the first layer hidden layer is represented by the following (j) formula:

[0027]

[0028] The activation value x2 of the second layer hidden layer is represented by the following (k) formula:

[0029] x2 = ReLU (W3x1 + b3) (k).

[0030] The global value of the current state represented by the following (1) formula:

[0031] V (E HSt ) = W V x2 + b V (l).

[0032] The advantage value A (E HSt ,a) is represented by the following (m) formula:

[0033] A (E HDt ,a) = W A x2 + b A (m).

[0034] The state Q value when action a is taken represented by the following (n) formula:

[0035]

[0036] Where a represents the action of the policy selection model selected policy, represents the action space dimension, represents all the selectable actions, represents the global value of the current state, represents the advantage value of each action under a given state; x1, x2 are the activation values of the two hidden layers, respectively; State Q value when action a is taken

[0037] When training, the two modules need to be trained together, and the parameters of the reward function fitting module are collectively referred to as φ, and the parameters of the policy selection module are collectively referred to as θ. The reward function fitting module uses the method of minimizing the KL divergence to fit the posterior distribution of the reward, and the corresponding loss function is KL_Loss. The policy selection module is trained using the negative log-likelihood loss method, a labelThe action strategy is correctly selected, and the corresponding loss function is NLL_Loss. In addition, R_implicit represents the reward size estimated by the strategy network in the process of state transition, Constrain_Loss represents the constraint loss, which is used to maintain the consistency between the learned strategy and the inferred reward distribution, thereby improving the optimization efficiency. The final optimization loss function is Total_Loss, and α and β are used to control the loss proportion.

[0038] The KL divergence loss KL_Loss is represented by the following formula (o):

[0039]

[0040] Wherein, mean represents the mean of the current reward function distribution output by RewardEncoder, and log_sds represents the logarithmic standard deviation.

[0041] The corresponding loss function NLL_Loss is represented by the following formula (p):

[0042]

[0043] Wherein, a label The action strategy is correctly selected, is the comprehensive state encoding of the dialogue history.

[0044] The reward size R_implicit is represented by the following formula (q):

[0045] R_implicit=Q(E HSi ,s i )-Q;E′ HSi ,s′ i ) (q).

[0046] Wherein, is the comprehensive state encoding of the dialogue history, s i represents the strategy action selection corresponding to the current dialogue round, represents the state encoding of the next round, s′ i represents the strategy action selection corresponding to the next round of dialogue.

[0047] The constraint loss Constrain_Loss is represented by the following formula (r):

[0048]

[0049] Wherein, R_implicit represents the reward value, represents a normal distribution.

[0050] The optimization loss function Total_Loss is represented by the following (s) formula:

[0051] Total_Loss = a·KL_Loss + b·NLL_Loss + Constrain_Loss (s).

[0052] Wherein, a and b are two different loss weights, KL_Loss represents the KL divergence loss, NLL_Loss represents the negative log-likelihood loss, and Constrain_Loss represents the constraint loss.

[0053] After training, the Q value corresponding to each action in the current state is obtained using the policy selection module, and the action with the maximum Q value is finally selected as the subsequent selection policy. Here, the action with the maximum Q value is finally selected as the action as the optimal action selection for the current dialogue state, that is, the corresponding policy s t .

[0054] The policy action a t is represented by the following (t) formula:

[0055]

[0056] Wherein, represents the dialogue state encoding, represents the candidate action set.

[0057] The dialogue common sense analysis and reasoning module is used to generate common sense analysis and prediction based on the current dialogue history. The Llama2-hf-7b model is fine-tuned using the dialogue history data and the common sense analysis data identified by ChatGPT, and the goal is to generate appropriate common sense knowledge and prediction according to the current dialogue. Wherein, the fine-tuning loss function Loss_commonsense is represented by the following (u) formula:

[0058]

[0059] Wherein, Z target generated content, C represents the input dialogue background prompt, and G represents the total length of the current target generated content; Z ＜g represents the content generated by the model;

[0060] The suggestion information z suggesion for the current user condition is represented by the following (w) formula:

[0061]

[0062] Wherein, represents a language model trained by LoRA adaptation, which is specially used to generate suggestions for the current dialogue condition, Csuggesion The inputted suggestion provides an adapted dialogue background prompt.

[0063] Current dialogue analysis of user's thought z intention is represented by the following (x) formula:

[0064]

[0065] wherein, is a LoRA adapted language model trained for parsing the relevant content of generating user's thought, C intention The inputted thought analysis adapted dialogue background prompt.

[0066] Model predicted subsequent occurrence event z subsequeJtvent is represented by the following (y) formula:

[0067]

[0068] wherein, is a LoRA adapted language model trained for predicting the subsequent occurrence event, C subsequentevent The inputted subsequent event prediction adapted dialogue background prompt.

[0069] wherein, Loss_commonsense is a common sense fine-tuning loss function, Z represents the total of common sense, and G represents the total length of current common sense data, Z ＜g represents the generated common sense content. The final obtained four LoRA model parameters fine-tuned, wherein represents a model for obtaining the current emotion of the user and the reason, and the current emotion of the user and the reason can be obtained by inputting the dialogue history z emotion ; z suggeestion represents the suggestion information for the current user's condition, which can be obtained by the model; z intention represents the possible intention of the model in the future reply, which can be obtained by ; z subsequentevent represents the condition that may occur in the subsequent dialogue, which can be obtained by .

[0070] The reply generation module is used for final emotional support dialogue reply generation, wherein the content Z collection is represented by the following (z) formula:

[0071] Z collecion = [z enotion ; z suggesion ; z intention ; z subsequentevent](z).

[0072] wherein Z collection represents the content set generated by the dialogue common sense reasoning analysis module, zemotion represents the current emotion of the user and the reason thereof; z suggestion represents the suggestion information for the current user condition; z intention represents the user's intention predicted by the model; z subsequentevent represents the condition that may occur after the dialogue.

[0073] The loss function Loss_response is represented by the following (I) formula:

[0074]

[0075] wherein Z collection represents the content set generated by the dialogue common sense reasoning analysis module, x represents the target generated reply; I represents the total length of the reply; x ＜i represents the generated reply; C t represents the background prompt for the generation of the tth round of dialogue; s t represents the text description of the selected strategy action; x represents the target generated reply; I represents the total length of the reply; x ＜i represents the generated reply. Using the trained model, in combination with the content Z collecion generated by the dialogue common sense reasoning analysis module and the selected dialogue strategy, the dialogue reply x t represented by the following (II) formula is obtained:

[0076]

[0077] wherein x t represents the dialogue reply of the tth round, s t represents the text description of the selected strategy action, Z collection represents the content set generated by the dialogue common sense reasoning analysis module; C t represents the background prompt for the generation of the tth round of dialogue.

[0078] Using the Llama2-hf-7b model, LoRA fine-tuning is performed, wherein the loss function is Loss_response, Z collection represents the content generated using the dialogue common sense reasoning analysis module, x represents the target generated reply, and I represents the total length of the reply, x ＜i represents the generated reply. Finally, using the trained model in combination with the previously generated common sense reasoning analysis content and the selected dialogue strategy, the dialogue reply x t .

[0079] The strategy emotional support dialogue task in step 1-1 selects a suitable dialogue strategy according to the current dialogue history, and the system converts the dialogue history together with the used dialogue strategy into a fixed code through a dialogue state encoder.

[0080] The dialogue strategy selection is performed using a variational Bayesian inverse reinforcement learning strategy selection module after obtaining the current dialogue state code in step 1-2. During the training process of the module, a neural network is used to fit the posterior distribution of the reward function, and demonstration data is used for policy selection training, while a constraint loss function is introduced to keep the reward function learning module consistent with the policy selection module.

[0081] The training in the dialogue common sense analysis reasoning module and the use of the dialogue common sense analysis module will analyze the current dialogue process and generate common sense from four angles.

[0082] The reply generation module splices the selected strategy and common sense information with the dialogue history to finally generate a suitable emotional support reply.

[0083] Compared with the prior art, the present application has the following beneficial technical effects and significant technical progress:

[0084] 1) Ease of use: the present application can accelerate model training, reduce application cost and time cost.

[0085] 2) Correctness: the present application designs a variational Bayesian inverse reinforcement learning strategy selection module and a common sense assisted framework, which can improve the accuracy of policy selection and effectively improve the information richness of generated replies.

[0086] 3) Practicality: the present application has wide practical significance and can be applied in real scenarios such as smart speakers, mobile phone intelligent assistants, etc., effectively improving user experience and providing suitable emotional support services for users. BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0088] The present application will be further described in detail below in combination with specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present application are the general knowledge and common sense in the art, and the present application has no special limitations.

[0089] Reference Figure 1This invention comprises an emotion support strategy selection module, a dialogue common sense analysis and reasoning module, and a response generation module. Through support strategy selection, dialogue common sense generation and analysis, it generates responses for the dialogue system. The specific process mainly includes the following steps:

[0090] Step 1: Predict the strategy needed to respond to the current conversation

[0091] In the policy-driven emotion support dialogue task, the system needs to select an appropriate policy to guide the dialogue based on the current dialogue history. To better select the dialogue policy, an emotion support policy selection module is proposed. This module consists of a dialogue state encoder and a variational Bayesian inverse reinforcement learning policy selection module, which selects the emotion support policy based on the current dialogue history information and the policies already used. First, the dialogue state encoder processes the dialogue history C into the input sequence H. t The text state is then input into the Roberta-base model for text state encoding, and the pooling layer output E is obtained. H Then perform a linear transformation to obtain E. HL Finally, the list of strategies used previously, S t By concatenating these elements, we obtain the comprehensive state code E of the current dialogue history. HSt The processing procedure is represented by the following (a) to (f):

[0092] C = (y0, x0, y1, x1, ..., y t (a);

[0093] H t =([CLS],y0,[SEP],x0,[SEP],y1,[SEP],x1,[SEP],…,y t (b);

[0094] S t =(s0,s1,s2,…,s t-1 (c);

[0095] E H =Roberta(H t [Pooler_Output] (d);

[0096] E HL =Linear(E H (e);

[0097] E HSt =[E HL S t (f).

[0098] E will follow HStThe input is fed into the variational Bayesian inverse reinforcement learning policy selection module for policy selection. This module uses the variational Bayesian inverse reinforcement learning method to train the policy selection module, which mainly includes a reward function fitting module and a policy selection module.

[0099] The implementation steps of the variational Bayesian inverse reinforcement learning method are as follows: First, the original dataset can be encoded using the dialogue state encoder mentioned above. Specifically, the construction method is as follows: For each complete dialogue in the dataset, data is constructed, with dialogue roles divided into helper and helper. The dialogue history before the helper's response is encoded using the dialogue state encoder, and then the response strategy used by the helper is added, thus obtaining sequence data D. raw , where m represents the number of complete dialogues in the dataset, and τ represents the maximum number of rounds in the current dialogue. Then D raw The data is further segmented to obtain training data in quadruplets, which contain the encoding of the previous dialogue's historical state, the strategy used by the helper in the previous state, the encoding of the next dialogue's historical state, and the strategy used by the helper in the next dialogue. n represents the final number of quadruplets obtained.

[0100] The sequence data D raw It can be expressed by the following formula (g):

[0101]

[0102] in, This represents the comprehensive state code of the t-th round of dialogue history of the i-th dialogue in the dataset. τ represents the strategy choice in the t-th round of the i-th dialogue, where t represents the current round number of the dialogue. I represents the maximum number of turns for the i-th dialogue in the current dataset, and m represents the total number of complete dialogues in the current dataset.

[0103] For D raw The training data D obtained by further segmentation is expressed by the following equation (h):

[0104]

[0105] in, This represents the total number of training data entries. D represents raw The comprehensive state encoding of the segmented dialogue history, s i D represents raw The usage strategies segmented from it This indicates that it is in the original dataset. The state encoding of the next round of dialogue, s′ i express Corresponding policy action selection.

[0106] The reward function fitting module includes a deep learning neural network RewardEncoder to learn the reward function variational posterior distribution, and sets the reward function as a Gaussian distribution, wherein the RewardEncoder is composed of an input value being a comprehensive state code of the dialogue history, W1, W2, b1 and b2 being trainable parameters, h1 representing an intermediate result vector, and finally outputting a vector with a dimension of 2, wherein the first dimension represents the mean of the current reward function distribution mean, and the second dimension represents the logarithmic standard deviation log_sds.

[0107] The intermediate result vector h1 is represented by the following formula (i):

[0108]

[0109] (mean,log_sds)=W2h1+b2(i)。

[0110] The policy selection module adopts the design method of Dueling Q-Network, wherein the input value is a comprehensive state code of the dialogue history, W3, W4, W V , W A , b3, b4, b V and b A are trainable parameters, a represents an action of the policy selected by the policy selection model, represents the dimension of the action space, represents the global value of the current state, A(E HSt ,a) represents the advantage value of each action under the given state, x1 and x2 are respectively the activation values of two hidden layers; is the state Q value when action a is taken

[0111] The activation value x1 of the first layer hidden layer is represented by the following formula (j):

[0112]

[0113] The activation value x2 of the second layer hidden layer is represented by the following formula (k):

[0114] x2=ReLU(W3x1+b3) (k)。

[0115] The global value of the current state is represented by the following formula (l):

[0116]

[0117] The advantage value A(EHSt a) represented by the following (m) formula:

[0118]

[0119] the state Q value when action a is taken represented by the following (n) formula:

[0120]

[0121] wherein a represents the action of the policy selected by the policy selection model, represents the dimension of the action space, represents all selectable actions, represents the global value of the current state, represents the advantage value of each action under a given state; x1 and x2 are respectively the activation values of two hidden layers; state Q value when action a is taken

[0122] When training, the two modules need to be trained together. The parameters of the reward function fitting module are collectively referred to as φ, and the parameters of the policy selection module are collectively referred to as θ. The reward function fitting module uses the method of minimizing the KL divergence to fit the posterior distribution of the reward, and the corresponding loss function is KL_Loss. The policy selection module is trained using the negative log-likelihood loss method, a label is the correct action policy, and the corresponding loss function is NLL_Loss. In addition, R_implicit represents the size of the reward implied in the process of state transition estimated by the policy network, and Constrain_Loss represents the constraint loss for maintaining the consistency between the learned policy and the inferred reward distribution, thereby improving the optimization efficiency. The final optimization loss function is Total_Loss, and α and β are used to control the loss proportion.

[0123] The KL divergence loss KL_Loss is represented by the following (o) formula:

[0124]

[0125] wherein mean represents the mean of the current reward function distribution output by RewardEncoder, and log_sds represents the log standard deviation.

[0126] The corresponding loss function NLL_Loss is represented by the following (p) formula:

[0127]

[0128] wherein alabel for the correct selection of action strategy, for the integrated state encoding of the dialogue history;

[0129] The reward size R_implicit is represented by the following (q) formula:

[0130]

[0131] wherein, s represents the integrated state encoding of the dialogue history, i s represents the policy action selection corresponding to the current dialogue round, s' represents the state encoding of the next round, i s' represents the policy action selection corresponding to the next round of dialogue

[0132] The constraint loss Constrain_Loss is represented by the following (r) formula:

[0133]

[0134] wherein, R_implicit represents the reward value, s represents the normal distribution.

[0135] The optimization loss function Total_Loss is represented by the following (s) formula:

[0136] Total_Loss = a·KL_Loss + b·NLL_Loss + Constrain_Loss (s).

[0137] wherein, a and b are two different loss weights respectively, KL_Loss represents the KL divergence loss, NLL_Loss represents the negative log-likelihood loss, and Constrain_Loss represents the constraint loss.

[0138] After the training is completed, the Q value corresponding to each action in the current state can be obtained using the policy selection module, and the action with the maximum Q value is finally selected as the subsequent selected policy. Here, the action with the maximum Q value is finally selected as the optimal action selection for the current dialogue state, that is, the corresponding policy s t .

[0139] The policy action a t is represented by the following (t) formula:

[0140]

[0141] wherein, s represents the dialogue state encoding, s represents the candidate action set.

[0142] Step two: establish a dialogue common sense analysis reasoning module, which is used to generate common sense analysis and prediction based on the current dialogue history; use the dialogue history data and the common sense analysis data identified by ChatGPT to fine-tune the Llama2-hf-7b model using LoRA, aiming to generate appropriate common sense knowledge and prediction according to the current dialogue. Among them, the fine-tuning loss function Loss_commonsense is represented by the following (u) formula:

[0143]

[0144] Among them, Ztarget generated content, C represents the input dialogue background prompt, and G represents the total length of the current target generated content; Z ＜g represents the content generated by the model.

[0145] User's current emotion and its reason z emotion represented by the following (v) formula:

[0146]

[0147] Among them, represents a LoRA-adapted language model trained to generate content related to the user's emotion and reason, C emotion represents the input emotion reason analysis extraction adapted dialogue background prompt.

[0148] Suggestion information for the current user's condition z suggestion represented by the following (w) formula:

[0149]

[0150] Among them, represents a LoRA-adapted language model trained to generate suggestions for the current dialogue condition, C suggestion represents the input suggestion providing adapted dialogue background prompt.

[0151] Analysis of the user's ideas in the current dialogue z intention represented by the following (x) formula:

[0152]

[0153] Among them, represents a LoRA-adapted language model trained to generate content related to the user's ideas, C intention represents the input idea analysis adapted dialogue background prompt.

[0154] Model-predicted subsequent events z subsequenteventis represented by the following (y) formula:

[0155]

[0156] wherein, represents a language model trained by LoRA adaptation, which is specially used to predict the generation of subsequent events, C subsequenevent represents the input of the subsequent event prediction adapted dialogue context prompt.

[0157] wherein, Loss_commonsense is a fine-tuning loss function, Z represents the total of common sense, G represents the total length of the current common sense data, Z ＜g represents the generated common sense content. Finally, four LoRA model parameters are obtained by fine-tuning, wherein represents a model for obtaining the current emotion of the user and the reason, which can obtain the current emotion of the user and the reason z emotion ; z suggestion represents the suggestion information for the current user condition, which can be obtained by model; z intention represents the possible intention of the model in the future reply, which can be obtained by ; z subsequentevent represents the possible situation that may occur in the subsequent dialogue, which is obtained by .

[0158] Step three: establish a reply generation module, which is used for the final emotional support dialogue reply generation, wherein the content Z collection is represented by the following (z) formula:

[0159] Z collection = [z emotion ; z suggesion ; z intention ; z subsequentevent ](z).

[0160] The current emotion of the user and the reason z emotion ; z suggestion represents the suggestion information for the current user condition; z intention represents the intention of the user predicted by the model; z subsequentevent represents the possible situation that may occur in the subsequent dialogue.

[0161] The loss function Loss_response is represented by the following (I) formula:

[0162]

[0163] wherein, Zcollection represents the content set generated by the dialogue common sense reasoning analysis module, x represents the target generated reply; I represents the total length of the reply; x ＜i represents the generated reply; C t represents the background prompt for the tth round of dialogue generation; s t represents the text description of the selected strategy action.

[0164] wherein x represents the target generated reply; I represents the total length of the reply; x ＜i represents the generated reply.

[0165] Using the trained model, in combination with the content Z generated by the dialogue common sense reasoning analysis module collection and the selected dialogue strategy, the dialogue reply x represented by the following (II) formula is obtained t :

[0166]

[0167] wherein x t represents the tth round of dialogue reply, s t represents the text description of the selected strategy action, Z collection represents the content set generated by the dialogue common sense reasoning analysis module; C t represents the background prompt for the tth round of dialogue generation.

[0168] The specific method is to use the Llama2-hf-7b model to perform LoRA fine-tuning, wherein the loss function is Loss_response, Z collection represents the content generated using the dialogue common sense reasoning analysis module, x represents the target generated reply, I represents the total length of the reply, x ＜i represents the generated reply. Finally, using the trained model in combination with the previously generated common sense reasoning analysis content and the selected dialogue strategy, the dialogue reply x t .

[0169] The above is only a further description of the present application and is not intended to limit the patent. Any equivalent implementation of the present application shall be included in the scope of the claims of the present patent.

Claims

1. An emotion-supporting dialogue system based on a variational Bayesian inverse reinforcement learning strategy, characterized in that, An emotion-supported dialogue system is constructed using an emotion support strategy selection module, a dialogue commonsense analysis and reasoning module, and a response generation module. Through support strategy selection, dialogue commonsense generation, and analysis, responses are generated for the dialogue system. The emotion support strategy selection module is a combination of a dialogue state encoder and a variational Bayesian inverse reinforcement learning strategy selection module. This module selects emotion support strategies based on current dialogue history information and already used strategies, specifically including: 1-1: The dialogue state encoder will record the dialogue history. and a list of previously used strategies The transformation into an eigenma matrix is ​​represented by the following (a) to (f): （a）； （b）； （c）； （d）； （e）； （f）； in, Indicates the current round number of the dialogue; This indicates the content previously responded by the model; This represents the user's history of statements; This indicates that the conversation history will be formatted appropriately. Will The input is fed into the Roberta-base model for encoding, and the output vector of the model's pooling layer is obtained. The input is then transformed into a linear layer to obtain the feature vector. With strategy list By concatenating the data, we obtain the comprehensive state code of the current dialogue history. ; 1-2: The variational Bayesian inverse reinforcement learning policy selection module predicts policies based on the current dialogue history. This module includes a reward function fitting module and a policy selection module, wherein the policy selection module is trained using the variational Bayesian inverse reinforcement learning method. The implementation steps of the variational Bayesian inverse reinforcement learning method are as follows: 1-2-1: Dialogue history encoding is performed on the original dataset using a dialogue state encoder. The specific construction method is as follows: For each complete dialogue in the dataset, data is constructed, with dialogue roles divided into helper and helper. The dialogue history before the helper's response is state-encoded using a dialogue state encoder, and then the response strategy used by the helper is added to obtain sequence data. , indicating the number of complete dialogues in the dataset. , indicates the maximum number of turns in the current dialogue. ; 1-2-2: Sequence data The data in the middle is split to obtain The training data for the tuples, wherein each tuple contains the encoding of the previous dialogue's historical state, the policy used by the helper in the previous state, the encoding of the next dialogue's historical state, and the policy used by the helper in the next state; the sequence data It can be expressed by the following formula (g): (g); in, The comprehensive state encoding of the t-th round dialogue history of the i-th dialogue; This refers to the strategy selection for the t-th round of the i-th dialogue; t is the current round number of the dialogue. is the maximum number of turns for the i-th dialogue in the current dataset; m is the total number of complete dialogues in the current dataset; For sequence data Further segmentation yields the training data as shown in equation (h) below. : （h）； in, This represents the total number of training data entries. for The comprehensive state encoding of the dialogue history segmented from the middle; for The strategy for using split-in; For the original dataset, it was in a central position The state encoding of the next round of dialogue; for Corresponding strategy action selection; 1-2-3: The reward function fitting module contains a deep learning neural network. To learn the variational posterior distribution of the reward function, we set the reward function to a Gaussian distribution, where... Its composition is as follows: 1) The input value is the comprehensive state code of the dialogue history. ,in , , and All of these are trainable parameters. This represents the intermediate result vector. The final reward function fitting module outputs a 2-dimensional vector, where the first dimension represents the mean of the current reward function distribution. The second dimension represents the logarithmic standard deviation. The intermediate result vector It can be expressed by the following equation (i): （i）； 2) Strategy Selection Module use The design approach uses a comprehensive state code of the dialogue history as input. , This represents the global value of the current state. This represents the advantage value of each action in a given state; , These are the activation values ​​for the two hidden layers, respectively. For state Taking action Q value at that time; Activation value of the first hidden layer It can be expressed by the following equation (j): （j）； Activation value of the second hidden layer It can be expressed by the following equation (k): （k）； The global value of the current state It can be expressed by the following equation (l): （l）； The advantage value It can be expressed by the following formula (m): （m）； The Q value It can be expressed by the following formula (n): （n）； The action for the strategy selected by the strategy selection model; Indicates the dimension of the action space; For all selectable actions; 3) The reward function fitting module and the policy selection module are trained collaboratively. The parameters of the reward function fitting module are collectively referred to as... The parameters of the strategy selection module are collectively referred to as The reward function fitting module uses minimizing the KL divergence to fit the posterior distribution of the reward; the policy selection module is trained using negative log-likelihood loss, with the corresponding loss function being... , This represents the implicit reward value of the Q-value estimated by the policy network during the state transition process. Let represent the constraint loss, and the final optimized loss function is: , and Used to control the proportion of losses; KL divergence loss It can be expressed by the following equation (o): （o）： in, for The mean of the current reward function distribution output; The standard deviation is the logarithm. It can be expressed by the following formula (p): （p）； in, To select the correct action strategy; A comprehensive state encoding for the dialogue history; The size of the reward It can be expressed by the following formula (q): （q）； in A comprehensive state encoding for the dialogue history; Select the strategy action corresponding to the current dialogue turn; Encode the state for the next round; Choose the strategy action for the next round of dialogue; The constraining loss It can be expressed by the following formula (r): （r）； in, This is the reward value; It follows a normal distribution; The optimization loss function It can be expressed by the following formula (s): （s）； in, and These are two different loss weights; For KL divergence loss; The loss is negative log-likelihood. For binding losses; 4) After training, use the policy selection module to calculate the Q-value for each action in the current state, select the action with the largest Q-value as the subsequent policy, and finally select the policy. The optimal action choice for the current dialogue state, i.e., the corresponding strategy. ; The strategy action It can be expressed by the following equation (t): （t）； in, Encode the dialogue state; For the candidate action set; The construction of the dialogue commonsense analysis and reasoning module specifically includes: 2-1: LoRA fine-tuning of the Llama2-hf-7b model using dialogue history data and commonsense analysis data identified by ChatGPT; wherein, the fine-tuning loss function... It can be expressed by the following formula (u): （u）； in, Generate content for the target; The input dialogue background prompt; G is the total length of the content generated for the current target; This refers to the content already generated by the model; User's current emotions and their causes It can be expressed by the following equation (v): （v）； in, A language model trained for LoRA, used to generate content related to user emotions and reasons; Extract appropriate dialogue background cues for analyzing the emotional causes of the input; Suggested information regarding the current user status It can be expressed by the following formula (w): （w）； in, A language model trained for LoRA, used to generate suggestions for the current dialogue situation; Provide appropriate dialogue context prompts for the input suggestions; Analysis of user thoughts in the current conversation It can be expressed by the following equation (x): （x）； in, A language model trained for LoRA, used to parse relevant content to generate user thoughts; Analyze and adapt dialogue context prompts to the input ideas; Model-predicted subsequent events This can be expressed by the following formula (y): （y）； in, A language model trained for LoRA is used to predict and generate subsequent events. Dialogue context cues adapted to predict subsequent events from the input; in, G represents the general term for common sense; G represents the total length of the current common sense data. This represents the common knowledge content that has already been generated; 2-2: Fine-tune using LoRA to obtain the user's current mood and its reasons. Suggested information regarding the current user status Possible intentions in future responses And what might happen after the conversation? Four LoRA model parameters, among which This represents a model for obtaining a user's current emotion and its cause. By inputting the user's dialogue history, it can obtain the user's current emotion and its cause. ; This indicates suggested information regarding the current user situation, through... Model acquisition; To represent the user's predicted intent, the model uses... To obtain; To indicate what might happen next in a conversation, use... To obtain; The construction of the response generation module specifically includes: 3-1: Using the four LoRA model parameters obtained after fine-tuning, an emotion-supported dialogue response generation module is constructed; among them, the content set generated by the dialogue commonsense reasoning analysis module... It can be expressed by the following equation (z): （z）； in, The user's current emotion and its cause can be obtained by inputting their conversation history. This provides suggested information regarding the current user situation; The user's intent as predicted by the model; This is to consider what might happen after the dialogue. loss function From the following ( The formula is expressed as: （ ）； in, The content collection generated for the dialogue common sense reasoning analysis module; Generate a response for the target; This is the total length of the reply; This is a response that has already been generated; Background cues generated for the t-th round of dialogue; A text description of the selected strategy action; 3-2: Using the training results The model, combined with the content generated by the dialogue common sense reasoning analysis module. And the selected dialogue strategy, resulting in the following ( Dialogue response expressed in the form of ) : （ ）； in, For the response in the t-th round of dialogue; A text description of the selected strategy action; The content collection generated for the dialogue common sense reasoning analysis module; Background prompts generated for the t-th round of dialogue.

2. The emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy according to claim 1, characterized in that, Step 1-1 selects an appropriate dialogue strategy based on the current dialogue history and converts the dialogue history along with the used dialogue strategy into a fixed code through a dialogue state encoder.

3. The emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy according to claim 1, characterized in that, After obtaining the current dialogue state encoding, steps 1-2 use the variational Bayesian inverse reinforcement learning policy selection module to select a dialogue policy. The training of this module uses a neural network to fit the posterior distribution of the reward function, and uses demonstration data for policy selection training. At the same time, a constraint loss function is introduced to ensure that the reward function fitting module and the policy selection module are consistent.

4. The emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy according to claim 1, characterized in that, The dialogue common sense analysis and reasoning module analyzes the user's current emotion and its reasons. Suggested information regarding the current user status Possible intentions in future responses And what might happen after the conversation? Four LoRA model parameters are used to analyze the current dialogue process and generate common sense.

5. The emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy according to claim 1, characterized in that, The response generation module combines the selected strategy, common sense information, and dialogue history to generate an emotionally supportive response.

Images