Emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy

Abstract

The invention discloses an emotional support dialogue system based on a variational Bayesian inverse reinforcement learning strategy, which is characterized by adopting a strategy emotional support dialogue system constructed by an emotional support strategy selection module, a dialogue common sense analysis and reasoning module and a reply generation module, the emotion support strategy selection module performs strategy selection through a dialogue state encoder and a variational Bayesian inverse reinforcement learning strategy selection module according to current dialogue historical information and used strategies; the dialogue common sense analysis and reasoning module generates common sense information analysis and prediction related to the current dialogue; and the reply generation module generates a reply according to the generated content and the dialogue history. Compared with the prior art, the method has the advantages that the information richness and accuracy of reply generation are enhanced, the strategy selection accuracy and the user experience in a strategy emotion support dialogue scene are effectively improved, and the method has good application prospects and commercial development value.

Description

Technical Field

[0001] The present invention relates to the technical field of strategic emotion support dialogue systems, and in particular to an emotion support dialogue system based on a variational Bayesian inverse reinforcement learning strategy. Background Art

[0002] Emotional support dialogue systems have recently attracted much attention due to people's daily mental health needs. Such systems interact with users, gradually understand the user's predicament in a guiding manner, provide users with appropriate psychological comfort, and finally propose appropriate solutions and solutions for users. Daily emotional support and psychological comfort are important aspects of maintaining mental health, which involves an individual's attitude and pressure towards life, work, social relationships, etc. At present, it is difficult for relatives, friends or professionals to understand an individual's emotional state in real time, because the communication time is limited and there is a lack of professional psychological knowledge, which makes the communication efficiency relatively low.

[0003] In order to provide better emotional support, an emotional support dialogue dataset ESConv with 8 support strategies annotated has been proposed in recent years. In this task, the dialogue system will select the appropriate emotional support strategy based on the current dialogue history. The basic framework of the emotional support strategy is the three-stage model of "exploration-understanding-action", so choosing the appropriate dialogue strategy will guide the generation of dialogue responses more reasonably.

[0004] The existing technology uses reinforcement learning to select strategies and manually sets the reward function as the future emotional changes of the help-seeker after responding to the strategy adopted by the current model. This process involves single-step user simulation, namely environment simulation. However, in actual emotional support tasks, emotional support dialogue datasets are highly professional, and their environments and task objectives are complex and difficult to simulate. In addition, there is a lack of expert strategy annotations in the process of exploring the environment. At the same time, the design of the reward function also needs to take into account the complex potential influencing factors in the dialogue scenario. Summary of the invention

[0005] The purpose of the present invention is to provide an emotion support dialogue system based on a variational Bayesian inverse reinforcement learning strategy in response to the deficiencies in the prior art. The emotion support dialogue system is constructed by an emotion support strategy selection module, a dialogue common sense analysis and reasoning module, and a reply generation module. The emotion support strategy selection module performs strategy selection based on the current dialogue history information and the strategies that have been used. By supporting strategy selection, dialogue common sense generation and analysis, the information richness of the replies generated by the dialogue system is improved, and the accuracy of strategy selection is greatly improved. This method effectively improves the accuracy of strategy selection and the user experience in the strategy emotion support dialogue scenario. It only needs to learn from the existing data set and learn the preferences of expert trajectories without manually setting the reward function. It can enhance the information richness and accuracy of the generated replies, and improve the accuracy of strategy selection. It has good application prospects and commercial development value.

[0006] The specific technical scheme for achieving the purpose of the present invention is: an emotional support dialogue system based on a variational Bayesian inverse reinforcement learning strategy, which is characterized by using an emotional support dialogue system constructed by an emotional support strategy selection module, a dialogue common sense analysis and reasoning module, and a reply generation module. The reply of the dialogue system is generated by supporting strategy selection, dialogue common sense generation and analysis. The emotional support strategy selection module is composed of a dialogue state encoder and a variational Bayesian inverse reinforcement learning strategy selection module. The module selects an emotional support strategy according to the current dialogue history information and the strategies that have been used, and specifically includes:

[0007] 1-1: The dialogue state encoder converts the dialogue history C and the previously used strategy list S into a feature matrix. The processing process is represented by the following (a) to (f):

[0008] C=(y0,x0,y,x,…,y t ) (a);

[0009] H t =([CLS],y0,[SEP],x0,[SEP],y,[SEP],x,[SEP],…,y t ) (b);

[0010] S t =(s0,s,s2,…,s t- ) (c);

[0011] E H =Roberta(H t )[Pooler_Output] (d);

[0012] E HL =Linear(E H ) (e);

[0013] E H+t =[E HL ; S t ] (f);

[0014] Among them, t represents the number of rounds in the current conversation; x represents the content of the model's previous reply; y represents the user's historical sentences; H t It means processing the conversation history into a suitable format, inputting it into the Roberta-base model for encoding, and obtaining the model's pooling layer output vector E H , and then input it into the linear layer for dimensional transformation to obtain E HL , and finally the strategy list S that has been used in the previous historical dialogue t After splicing, we get the comprehensive state code E of the current dialogue history HSt .

[0015] 1-2: The variational Bayesian inverse reinforcement learning strategy selection module predicts the strategy based on the current dialogue history. In this module, the variational Bayesian inverse reinforcement learning method is used to train the strategy selection module, which mainly includes the reward function fitting module and the strategy selection module. The implementation steps of the variational Bayesian inverse reinforcement learning method are as follows: First, the original data set can be encoded with the dialogue state encoder mentioned above. The specific construction method is as follows: for each complete dialogue in the data set, data is constructed. The dialogue roles are divided into helpers and helpers. The dialogue history before the helper's reply is state-encoded using the dialogue state encoder, and then the reply strategy used by the helper is added to obtain the sequence data D raw , m represents the number of complete conversations in the data set, and τ represents the maximum number of rounds in the current conversation. Then D raw The data is further segmented, and the training data obtained is a four-tuple, which includes the encoding of the previous dialogue history state, the strategy used by the helper in the previous state, the encoding of the subsequent dialogue history state and the strategy used by the subsequent helper in the same dialogue. n represents the number of four-tuples finally obtained.

[0016] The sequence data D raw It is expressed by the following formula (g):

[0017]

[0018] in, represents the comprehensive state encoding of the t-th round of dialogue history of the ith dialogue in the dataset, represents the strategy choice of the tth round of dialogue in the ith dialogue, t represents the current round number of the dialogue, τ IIt represents the maximum number of rounds of the ith conversation in the current data set, and m represents the number of complete conversations contained in the current data set.

[0019] Right D raw The training data D obtained by further segmentation is expressed by the following formula (h):

[0020]

[0021] in, Represents the total number of training data. Indicates D raw The comprehensive state encoding of the segmented dialogue history, s i Indicates D raw The usage strategies that are separated out are: Indicates that in the original data set The state encoding of the next round of dialogue, s′ i express Corresponding strategy action selection.

[0022] The reward function fitting module contains a deep learning neural network RewardEncoder to learn the variational posterior distribution of the reward function. The reward function is set to Gaussian distribution. The composition of RewardEncoder is as follows: the input value is the comprehensive state encoding of the dialogue history, W1, W2, b1 and b2 are all trainable parameters, h1 represents the intermediate result vector, and finally the module outputs a vector of dimension 2, where the first dimension represents the mean of the current reward function distribution, and the second dimension represents the logarithmic standard deviation log_sds. The intermediate result vector h1 is represented by the following formula (i):

[0023] h1=ReLU(W1E HSt +b1)

[0024] (mean,log_sds)=W2h1+b2 (i);

[0025] The strategy selection module adopts the Dueling Q-Network design, where the input value is the comprehensive state encoding of the dialogue history, W3, W4, W V , W A , b3, b4, b V and b A is a trainable parameter, a represents the action of the strategy selected by the strategy selection model, represents the action space dimension, Represents the global value of the current state, represents the advantage value of each action in a given state, and x1 and x2 are the activation values ​​of the two hidden layers respectively; Status The Q value when taking action a;

[0026] The activation value x1 of the first hidden layer is expressed by the following formula (j):

[0027]

[0028] The activation value x2 of the second hidden layer is expressed by the following formula (k):

[0029] x2=ReLU(W3x1+b3) (k).

[0030] The global value of the current state It is expressed by the following formula (1):

[0031] V(E HSt )=W V x2+b V (l).

[0032] The advantage value A(E HSt ,a) is expressed by the following formula (m):

[0033] A(E HDt ,a)=W A x2+b A (m).

[0034] The status The Q value when taking action a It is expressed by the following formula (n):

[0035]

[0036] Among them, a represents the action of the strategy selected by the strategy selection model, represents the action space dimension, Indicates all the available actions. Represents the global value of the current state, represents the advantage value of each action in a given state; x1 and x2 are the activation values ​​of the two hidden layers respectively; Status The Q value when taking action a;

[0037] During training, these two modules need to be trained together. The parameters of the reward function fitting module are collectively referred to as φ, and the parameters of the strategy selection module are collectively referred to as θ. The reward function fitting module uses the method of minimizing KL divergence to fit the posterior distribution of the reward, corresponding to the loss function KL_Loss. The strategy selection module is trained using the negative log-likelihood loss method. labelFor the correct action strategy, the corresponding loss function is NLL_Loss. In addition, R_implicit represents the implicit reward size of the Q value estimated by the policy network during the state transition process, and Constrain_Loss represents the constraint loss, which is used to maintain the consistency between the learned strategy and the inferred reward distribution, thereby improving the optimization efficiency. The final optimization loss function is Total_Loss, and α and β are used to control the loss ratio.

[0038] The KL divergence loss KL_Loss is expressed by the following formula (o):

[0039]

[0040] Among them, mean represents the mean of the current reward function distribution output by RewardEncoder, and log_sds represents the logarithmic standard deviation.

[0041] The corresponding loss function NLL_Loss is expressed by the following formula (p):

[0042]

[0043] Among them, a label To select the correct action strategy, Encodes the comprehensive state of the conversation history.

[0044] The reward size R_implicit is expressed by the following formula (q):

[0045] R_implicit=Q(E HSi ,s i )-Q; E′ HSi ,s′ i ) (q).

[0046] in, Represents the comprehensive state encoding of the conversation history, s i Indicates the strategic action selection corresponding to the current dialogue turn, Represents the state code of the next round, s′ i Indicates the strategic action selection corresponding to the next round of dialogue.

[0047] The constraint loss Constrain_Loss is expressed by the following formula (r):

[0048]

[0049] Among them, R_implicit represents the reward value, Represents a normal distribution.

[0050] The optimization loss function Total_Loss is expressed by the following formula (s):

[0051] Total_Loss=α·KL_Loss+β·NLL_Loss+Constrain_Loss(s).

[0052] Among them, α and β are two different loss weights, KL_Loss represents KL divergence loss, NLL_Loss represents negative log-likelihood loss, and Constrain_Loss represents constraint loss.

[0053] After training is completed, the strategy selection module is used to obtain the Q value corresponding to each action in the current state, and finally the action with the largest Q value is selected as the strategy for subsequent selection. As the optimal action choice for the current dialogue state, that is, the corresponding strategy s t .

[0054] The policy action a t It is expressed by the following (t) formula:

[0055]

[0056] in, Indicates the dialogue state code, Represents a set of candidate actions.

[0057] The conversation common sense analysis and reasoning module is used to generate common sense analysis and prediction based on the current conversation history; the Llama2-hf-7b model is fine-tuned using the conversation history data and the common sense analysis data identified by ChatGPT, with the goal of generating appropriate common sense knowledge and predictions based on the current conversation. The fine-tuning loss function Loss_commonsense is expressed by the following (u) formula:

[0058]

[0059] Where Z is the target generated content, C is the input dialogue background prompt, and G is the total length of the current target generated content; ＜g Represents the content that the model has generated;

[0060] Suggested information for the current user situation suggesion It is expressed by the following formula (w):

[0061]

[0062] in, represents the trained LoRA-adapted language model, which is specifically used to generate suggestions for the current dialogue situation, Csuggesion The input suggestions provide appropriate dialog context prompts.

[0063] Analysis of the user's thoughts in the current conversation intention It is expressed by the following formula (x):

[0064]

[0065] in, represents the trained LoRA-adapted language model, which is specially used to parse and generate relevant content of user ideas. intention Indicates the conversation context prompts adapted to the input idea analysis.

[0066] The subsequent event z predicted by the model subsequeJtvent It is expressed by the following formula (y):

[0067]

[0068] in, represents the trained LoRA-adapted language model, which is specifically used to predict and generate subsequent events. subsequentevent Indicates the conversational context prompts adapted to the input's subsequent event predictions.

[0069] Among them, Loss_commonsense fine-tunes the loss function, Z represents the general term for common sense, G represents the total length of the current common sense data, and Z ＜g Represents the generated common sense content. The final fine-tuned four LoRA model parameters are Represents a model for obtaining the user's current emotions and reasons. By inputting the conversation history, the user's current emotions and reasons can be obtained. emotion ;z suggeestion Indicates the recommended information for the current user status, which can be specified through Model acquisition; z intention To indicate the model's possible intention in future responses, you can use To obtain; z subsequentevent To indicate what might happen next in a conversation, use To obtain.

[0070] The reply generation module is used to generate the final emotional support dialogue reply, where the content Z generated by the dialogue common sense reasoning analysis module collection It is expressed by the following formula (z):

[0071] Z collecion =[z enotion ;z suggesion ;z intention ;z subsequentevent](z).

[0072] Among them, Z collection represents the content set generated by the dialogue common sense reasoning analysis module, zemotion Indicates the user's current emotions and their reasons; suggestion Indicates the recommended information for the current user status; intention represents the user's intention predicted by the model; subsequentevent Indicates what might happen next in the conversation.

[0073] The loss function Loss_response is expressed by the following formula (I):

[0074]

[0075] Among them, Z collection represents the content set generated by the dialogue common sense reasoning analysis module, x represents the target generated response; I represents the total length of the response; x ＜i Indicates a generated reply; C t represents the background prompts used for the t-th round of dialogue generation; s t represents the text description of the selected policy action; x represents the target generated response; I represents the total length of the response; x ＜i Indicates the generated response. Model, combined with the content Z generated by the dialogue common sense reasoning analysis module collecion And the selected dialogue strategy, the dialogue response x is expressed by the following formula (II) t :

[0076]

[0077] Among them, x t represents the dialogue response in round t, s t Represents the text description of the selected policy action, Z collection represents the content set generated by the dialogue common sense reasoning analysis module; C t Represents the background prompts used for the t-th round of dialogue generation.

[0078] Use the Llama2-hf-7b model to fine-tune LoRA, where the loss function is Loss_response, Z collection represents the content generated by the dialogue common sense reasoning analysis module, x represents the target generated response, I represents the total length of the response, and x ＜i Indicates the generated response. Finally, using the training The model combines the previously generated common sense reasoning analysis content and the selected dialogue strategy to obtain the dialogue response x t .

[0079] In the strategy emotion support dialogue task in step 1-1, a suitable dialogue strategy is selected according to the current dialogue history, and the system converts the dialogue history together with the used dialogue strategy into a fixed code through a dialogue state encoder.

[0080] In step 1-2, after obtaining the current dialogue state encoding, the variational Bayesian inverse reinforcement learning strategy selection module will be used to select the dialogue strategy. During the training process of the module, a neural network is used to fit the posterior distribution of the reward function, and the strategy selection training is performed using demonstration data. At the same time, a constrained loss function is introduced to keep the reward function learning module and the strategy selection module consistent.

[0081] The training in the dialogue common sense analysis and reasoning module and the use of the dialogue common sense analysis module will analyze the current dialogue process from four perspectives and generate common sense.

[0082] The response generation module combines the selected strategy and common sense information with the conversation history to ultimately generate a suitable emotional support response.

[0083] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0084] 1) Ease of use: The present invention can speed up model training and reduce application cost and time cost.

[0085] 2) Correctness: The present invention designs a variational Bayesian inverse reinforcement learning strategy selection module and a common sense-assisted framework, which can improve the accuracy of strategy selection and effectively improve the information richness of generated responses.

[0086] 3) Practicality: The present invention has broad practical significance and can be applied in real scenarios, such as smart speakers, mobile phone smart assistants, etc., to effectively improve user experience and provide users with appropriate emotional support services. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0088] The present invention is further described in detail below in conjunction with specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0089] See also Figure 1The present invention is composed of an emotional support strategy selection module, a dialogue common sense analysis and reasoning module, and a reply generation module. Through support strategy selection, dialogue common sense generation and analysis, the reply of the dialogue system is generated. The specific process mainly includes the following steps:

[0090] Step 1: Predict the strategy needed to respond to the current conversation

[0091] In the strategic emotion support dialogue task, the system needs to select the appropriate strategy to guide the dialogue according to the current dialogue history. In order to better select the dialogue strategy, an emotion support strategy selection module is proposed. This module is composed of a dialogue state encoder and a variational Bayesian inverse reinforcement learning strategy selection module. It selects the emotion support strategy according to the current dialogue history information and the strategies that have been used. First, the dialogue state encoder is used to process the dialogue history C into the input sequence H. t , input it into the Roberta-base model for text state encoding, and obtain its pooling layer output E H , and then linearly transform it to get E HL , the last and previously used strategy list S t Concatenate to get the comprehensive state code E of the current conversation history HSt The processing process is represented by the following (a) to (f):

[0092] C=(y0,x0,y1,x1,…,y t ) (a);

[0093] H t =([CLS],y0,[SEP],x0,[SEP],y1,[SEP],x1,[SEP],…,y t ) (b);

[0094] S t =(s0,s1,s2,…,s t-1 ) (c);

[0095] E H =Roberta(H t )[Pooler_Output] (d);

[0096] E HL =Linear(E H ) (e);

[0097] E HSt =[E HL ; S t ] (f).

[0098] Later, E HStThe data is input into the variational Bayesian inverse reinforcement learning strategy selection module for strategy selection. In this module, the variational Bayesian inverse reinforcement learning method is used to train the strategy selection module, which mainly includes the reward function fitting module and the strategy selection module.

[0099] The implementation steps of the variational Bayesian inverse reinforcement learning method are as follows: first, the original data set can be encoded with the dialogue state encoder mentioned above. The specific construction method is as follows: for each complete dialogue in the data set, data is constructed. The dialogue roles are divided into the helper and the helper. The dialogue history before the helper's reply is state-encoded using the dialogue state encoder, and then the reply strategy used by the helper is added to obtain the sequence data D raw , m represents the number of complete conversations in the data set, and τ represents the maximum number of rounds in the current conversation. Then D raw The data is further segmented, and the training data obtained is a four-tuple, which includes the encoding of the previous dialogue history state, the strategy used by the helper in the previous state, the encoding of the subsequent dialogue history state and the strategy used by the subsequent helper in the same dialogue. n represents the number of four-tuples finally obtained.

[0100] The sequence data D raw It is expressed by the following formula (g):

[0101]

[0102] in, represents the comprehensive state encoding of the t-th round of dialogue history of the ith dialogue in the dataset, represents the strategy choice of the tth round of dialogue in the ith dialogue, t represents the current round number of the dialogue, τ I It represents the maximum number of rounds of the ith conversation in the current data set, and m represents the number of complete conversations contained in the current data set.

[0103] Right D raw The training data D obtained by further segmentation is expressed by the following formula (h):

[0104]

[0105] in, Represents the total number of training data. Indicates D raw The comprehensive state encoding of the segmented dialogue history, s i Indicates D raw The usage strategies that are separated out are: Indicates that in the original data set The state encoding of the next round of dialogue, s′ i express Corresponding strategy action selection.

[0106] The reward function fitting module includes a deep learning neural network RewardEncoder to learn the variational posterior distribution of the reward function, and sets the reward function to Gaussian distribution. The composition of RewardEncoder is as follows: the input value is the comprehensive state encoding of the dialogue history, W1, W2, b1 and b2 are all trainable parameters, h1 represents the intermediate result vector, and finally the module outputs a vector of dimension 2, in which the first dimension represents the mean mean of the current reward function distribution, and the second dimension represents the logarithmic standard deviation log_sds.

[0107] The intermediate result vector h1 is expressed by the following formula (i):

[0108]

[0109] (mean,log_sds)=W2h1+b2(i).

[0110] The strategy selection module adopts the design of Dueling Q-Network, where the input value is the comprehensive state encoding of the dialogue history, W3, W4, W V , W A , b3, b4, b V and b A is a trainable parameter, a represents the action of the strategy selected by the strategy selection model, represents the action space dimension, Indicates the global value of the current state, A(E HSt ,a) represents the advantage value of each action in a given state, x1 and x2 are the activation values ​​of the two hidden layers respectively; Status The Q value when taking action a

[0111] The activation value x1 of the first hidden layer is expressed by the following formula (j):

[0112]

[0113] The activation value x2 of the second hidden layer is expressed by the following formula (k):

[0114] x2=ReLU(W3x1+b3) (k).

[0115] The global value of the current state It is expressed by the following formula (1):

[0116]

[0117] The advantage value A(EHSt ,a) is expressed by the following formula (m):

[0118]

[0119] The status The Q value when taking action a It is expressed by the following formula (n):

[0120]

[0121] Among them, a represents the action of the strategy selected by the strategy selection model, represents the action space dimension, Indicates all the available actions. Represents the global value of the current state, represents the advantage value of each action in a given state; x1 and x2 are the activation values ​​of the two hidden layers respectively; Status The Q value when taking action a.

[0122] During training, these two modules need to be trained together. The parameters of the reward function fitting module are collectively referred to as φ, and the parameters of the strategy selection module are collectively referred to as θ. The reward function fitting module uses the method of minimizing KL divergence to fit the posterior distribution of the reward, corresponding to the loss function KL_Loss. The strategy selection module is trained using the negative log-likelihood loss method. label For the correct action strategy, the corresponding loss function is NLL_Loss. In addition, R_implicit represents the implicit reward size of the Q value estimated by the policy network during the state transition process, and Constrain_Loss represents the constraint loss, which is used to maintain the consistency between the learned strategy and the inferred reward distribution, thereby improving the optimization efficiency. The final optimization loss function is Total_Loss, and α and β are used to control the loss ratio.

[0123] The KL divergence loss KL_Loss is expressed by the following formula (o):

[0124]

[0125] Among them, mean represents the mean of the current reward function distribution output by RewardEncoder, and log_sds represents the logarithmic standard deviation.

[0126] The corresponding loss function NLL_Loss is expressed by the following formula (p):

[0127]

[0128] Among them, alabel To select the correct action strategy, Encode the comprehensive state of the conversation history;

[0129] The reward size R_implicit is expressed by the following formula (q):

[0130]

[0131] in, Represents the comprehensive state encoding of the conversation history, s i Indicates the strategic action selection corresponding to the current dialogue turn, Represents the state code of the next round, s′ i Indicates the strategy action selection corresponding to the next round of dialogue

[0132] The constraint loss Constrain_Loss is expressed by the following formula (r):

[0133]

[0134] Among them, R_implicit represents the reward value, Represents a normal distribution.

[0135] The optimization loss function Total_Loss is expressed by the following formula (s):

[0136] Total_Loss=α·KL_Loss+β·NLL_Loss+Constrain_Loss(s).

[0137] Among them, α and β are two different loss weights, KL_Loss represents KL divergence loss, NLL_Loss represents negative log-likelihood loss, and Constrain_Loss represents constraint loss.

[0138] After training is completed, the strategy selection module can be used to obtain the Q value corresponding to each action in the current state, and finally select the action with the largest Q value as the subsequent strategy. As the optimal action choice for the current dialogue state, that is, the corresponding strategy s t .

[0139] The policy action a t It is expressed by the following (t) formula:

[0140]

[0141] in, Indicates the dialogue state code, Represents a set of candidate actions.

[0142] Step 2: Establish a dialogue common sense analysis and reasoning module, which is used to generate common sense analysis and predictions based on the current dialogue history; use the dialogue history data and the common sense analysis data identified by ChatGPT to perform LoRA fine-tuning on the Llama2-hf-7b model, with the goal of generating appropriate common sense knowledge and predictions based on the current dialogue. Among them, the fine-tuning loss function Loss_commonsense is expressed by the following (u) formula:

[0143]

[0144] Where Z is the target generated content, C is the input dialogue background prompt, and G is the total length of the current target generated content; ＜g Represents content that has been generated by the model.

[0145] The user's current emotion and the reason for it emotion It is expressed by the following formula (v):

[0146]

[0147] in, represents the trained LoRA-adapted language model, which is specifically used to generate relevant content about user emotions and reasons. emotion The emotional reason analysis of the input extracts the appropriate conversation context prompts.

[0148] Suggested information for the current user situation suggestion It is expressed by the following formula (w):

[0149]

[0150] in, represents the trained LoRA-adapted language model, which is specifically used to generate suggestions for the current dialogue situation, C suggestion The input suggestions provide appropriate dialog context prompts.

[0151] Analysis of the user's thoughts in the current conversation intention It is expressed by the following formula (x):

[0152]

[0153] in, represents the trained LoRA-adapted language model, which is specially used to parse and generate relevant content of user ideas. intention Indicates the conversation context prompts adapted to the input idea analysis.

[0154] The subsequent event z predicted by the model subsequenteventIt is expressed by the following formula (y):

[0155]

[0156] in, represents the trained LoRA-adapted language model, which is specifically used to predict and generate subsequent events. subsequenevent Indicates the conversational context prompts adapted to the input's subsequent event predictions.

[0157] Among them, Loss_commonsense fine-tunes the loss function, Z represents the general term for common sense, G represents the total length of the current common sense data, and Z ＜g Represents the generated common sense content. The final fine-tuned four LoRA model parameters are Represents a model for obtaining the user's current emotions and reasons. By inputting the conversation history, the user's current emotions and reasons can be obtained. emotion ;z suggestion Indicates the recommended information for the current user status, which can be specified through Model acquisition; z intention To indicate the model's possible intention in future responses, you can use To obtain; z subsequentevent To indicate what might happen next in a conversation, use To obtain.

[0158] Step 3: Establish a response generation module, which is used to generate the final emotional support dialogue response. The content Z generated by the dialogue common sense reasoning analysis module collection It is expressed by the following formula (z):

[0159] Z collection =[z emotion ;z suggesion ;z intention ;z subsequentevent ](z).

[0160] By inputting the conversation history, we can get the user’s current emotion and its reason. emotion ;z suggestion Indicates the recommended information for the current user status; intention represents the user's intention predicted by the model; subsequentevent Indicates what might happen next in the conversation.

[0161] The loss function Loss_response is expressed by the following formula (I):

[0162]

[0163] Among them, Zcollection represents the content set generated by the dialogue common sense reasoning analysis module, x represents the target generated response; I represents the total length of the response; x ＜i Indicates a generated reply; C t represents the background prompts used for the t-th round of dialogue generation; s t Indicates a text description of the selected policy action.

[0164] Where x represents the target generated response; I represents the total length of the response; x ＜i Indicates a reply that has been generated.

[0165] Use the trained Model, combined with the content Z generated by the dialogue common sense reasoning analysis module collection And the selected dialogue strategy, the dialogue response x is expressed by the following formula (II) t :

[0166]

[0167] Among them, x t represents the dialogue response in round t, s t Represents the text description of the selected policy action, Z collection represents the content set generated by the dialogue common sense reasoning analysis module; C t Represents the background prompts used for the t-th round of dialogue generation.

[0168] The specific method is to use the Llama2-hf-7b model to fine-tune LoRA, where the loss function is Loss_response, Z collection represents the content generated by the dialogue common sense reasoning analysis module, x represents the target generated response, I represents the total length of the response, and x ＜i Indicates the generated response. Finally, using the training The model combines the previously generated common sense reasoning analysis content and the selected dialogue strategy to obtain the dialogue response x t .

[0169] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. An emotional support dialogue system based on variational Bayesian inverse reinforcement learning strategy, characterized in that: The emotional support dialogue system is constructed by an emotional support strategy selection module, a dialogue common sense analysis and reasoning module, and a response generation module. The response of the dialogue system is generated by supporting strategy selection, dialogue common sense generation and analysis. The emotional support strategy selection module is composed of a dialogue state encoder and a variational Bayesian inverse reinforcement learning strategy selection module. The module selects emotional support strategies according to the current dialogue history information and the strategies that have been used, specifically including: 1-1: The dialogue state encoder converts the dialogue history C and the previously used strategy list S t Converted into a feature matrix, the processing process is expressed as follows (a) to (f): C=(y0,x0,y1,x1,…,y t ) (a); H t =([CLS],y0,[SEP],x0,[SEP],y1,[SEP],x1,[SEP],…,y t ) (b); S t =(s0,s1,s2,…,s t-1 ) (c); E H =Roberta(H t )[Pooler_Output] (d); AND HL =Linear(E H ) (And); Among them, t represents the number of rounds in the current conversation; x represents the content of the model's previous reply; y represents the user's historical sentences; H t Indicates processing of the conversation history into a suitable format; H t Input to the Roberta-base model for encoding and obtain the model's pooling layer output vector E H Then input the linear layer for dimension conversion, and the obtained feature vector E HL With strategy list S t Splice to get the comprehensive state encoding of the current conversation history 1-2: The variational Bayesian inverse reinforcement learning strategy selection module predicts the strategy based on the current dialogue history. The variational Bayesian inverse reinforcement learning strategy selection module includes: a reward function fitting module and a strategy selection module, wherein the strategy selection module is trained using the variational Bayesian inverse reinforcement learning method; the implementation steps of the variational Bayesian inverse reinforcement learning method are as follows: 1-2-1: Use the dialogue state encoder to encode the dialogue history on the original data set. The specific construction method is as follows: for each complete dialogue in the data set, data is constructed. The dialogue roles are divided into help seekers and helpers. The dialogue history before the helper's reply is state-encoded using the dialogue state encoder, and then the reply strategy used by the helper is added to obtain the sequence data D raw , represents the number of complete conversations in the data set m, represents the maximum number of rounds of the current conversation τ; 1-2-2: Sequence data D raw The data in the sequence data D are segmented to obtain n-tuple training data, wherein the four-tuple includes the previous dialogue history state code, the strategy used by the previous state helper, the subsequent dialogue history state code and the strategy used by the subsequent helper in the same dialogue; the sequence data D raw It is expressed by the following formula (g): in, Encode the comprehensive state of the t-th round of dialogue history of the ith dialogue; is the strategy choice for the tth round of dialogue in the ith dialogue; t is the current round number of the dialogue; τ i is the maximum number of rounds of the ith conversation in the current data set; m is the number of complete conversations contained in the current data set; For sequence data D raw Further segmentation yields the training data D shown in the following formula (h): in, is the total number of training data; D raw The comprehensive state encoding of the dialogue history segmented from the i D raw The usage strategy of the middle segmentation; The original data set is in The state encoding of the next round of dialogue; s′ i for Corresponding strategic action selection; 1-2-3: The reward function fitting module contains a deep learning neural network RewardEncoder to learn the variational posterior distribution of the reward function. The reward function is set to Gaussian distribution. The composition of RewardEncoder is as follows: 1) The input value is the comprehensive state encoding of the conversation history Where W1, W2, b1 and b2 are all trainable parameters, h1 represents the intermediate result vector, and the final reward function fitting module outputs a vector of dimension 2, where the first dimension represents the mean of the current reward function distribution, and the second dimension represents the logarithmic standard deviation log_sds; the intermediate result vector h1 is represented by the following formula (i): h1=ReLU(W1E HSt +b1) (mean,log_sds)=W2h1+b2 (i); 2) Strategy selection module The Dueling Q-Network design is adopted, where the input value is the comprehensive state encoding of the conversation history. Represents the global value of the current state, represents the advantage value of each action in a given state; x1 and x2 are the activation values ​​of the two hidden layers respectively; Status The Q value when taking action a; The activation value x1 of the first hidden layer is expressed by the following formula (j): The activation value x2 of the second hidden layer is expressed by the following formula (k): x2=ReLU(W3x1+b3) (k); The global value V(E) of the current state HSt ) is expressed by the following formula (1): V(E HSt )=W V x2+b V (l); The advantage value A(E HSt ,a) is expressed by the following formula (m): A(E HSt ,a)=W A x2+b A (m); The Q value Q(E HSt ,a) is expressed by the following formula (n): Among them, a is the action of the strategy selected by the strategy selection model; represents the action space dimension; for all selectable actions; 3) The reward function fitting module and the strategy selection module are trained in coordination, wherein the parameters of the reward function fitting module are collectively referred to as φ, and the parameters of the strategy selection module are collectively referred to as θ. The reward function fitting module uses the posterior distribution of the reward to be fitted by minimizing the KL divergence; the strategy selection module is trained using negative log-likelihood loss, and the corresponding loss function is NLL_Loss, R_implicit represents the implicit reward size of the Q value estimated by the strategy network during the state transition process, Constrain_Loss represents the constraint loss, and the final optimization loss function is Total_Loss, α and β are used to control the loss ratio; The KL divergence loss KL_Loss is expressed by the following formula (o): Among them, mean is the mean of the current reward function distribution output by RewardEncoder; log_sds is the logarithmic standard deviation; The corresponding negative log-likelihood loss NLL_Loss is expressed by the following formula (p): Among them, a label To select the correct action strategy; Encode the comprehensive state of the conversation history; The reward size R_implicit is expressed by the following formula (q): in, Encodes the comprehensive state of the conversation history; i Select the strategy action corresponding to the current dialogue round; Encode the state of the next round; s′ i Select the corresponding strategic action for the next round of dialogue; The constraint loss Constrain_Loss is expressed by the following formula (r): Among them, R_implicit is the reward value; is a normal distribution; The optimization loss function Total_Loss is expressed by the following formula (s): Total_Loss=α·KL_Loss+β·NLL_Loss+Constrain_Loss(s); Among them, α and β are two different loss weights; KL_Loss is KL divergence loss; NLL_Loss is negative log-likelihood loss; Constrain_Loss is constraint loss; 4) After training is completed, use the strategy selection module to obtain the Q value corresponding to each action in the current state, select the action with the largest Q value as the subsequent strategy, and finally select the strategy action As the optimal action choice for the current dialogue state, the corresponding strategy s t ; The policy action a t It is expressed by the following (t) formula: in, Encode the state of the conversation; is the candidate action set; The construction of the dialogue common sense analysis and reasoning module specifically includes: 2-1: Use the conversation history data and the common sense analysis data identified by ChatGPT to fine-tune the Llama2-hf-7b model with LoRA; the fine-tuning loss function Loss_commonsense is expressed by the following (u) formula: Among them, Z is the target generated content; C is the input dialogue background prompt; G is the total length of the current target generated content; Z ＜g The content that has been generated for the model; The user's current emotion and the reason for it emotion It is expressed by the following formula (v): in, is a trained LoRA-adapted language model, which is used to generate relevant content about user emotions and reasons; C emotion Extract appropriate conversation context prompts for input emotional reason analysis; Suggested information for the current user situation suggestion It is expressed by the following formula (w): in, is a trained LoRA-adapted language model, which is used to generate suggestions for the current dialogue situation; C suggestion Provide appropriate conversation context prompts for input suggestions; Analysis of the user's thoughts in the current conversation intention It is expressed by the following formula (x): in, C is a language model adapted from the trained LoRA, which is used to parse and generate relevant content of user ideas; intention Analyze the conversation context prompts that are appropriate for the input ideas; The subsequent event z predicted by the model subsequentevent It is expressed by the following formula (y): in, is a language model adapted by the trained LoRA, which is used to predict and generate subsequent events; C subsepuentevent Predicting appropriate conversation context cues for subsequent events of the input; Among them, Z represents the general term for common sense; G represents the total length of the current common sense data; Z ＜g Indicates the common sense content that has been generated; 2-2: Through LoRA fine-tuning, get the user's current emotions and their reasons emotion , Suggested information for the current user status suggestion , possible intentions in future responses intention What might happen next in the conversation? subsequentevnt Four LoRA model parameters, among which Represents a model for obtaining the user's current emotions and reasons. By inputting the conversation history, the user's current emotions and reasons can be obtained. emotion ;z suggestion Indicates the recommended information for the current user situation, through Model acquisition; z intention Indicates the user's intention predicted by the model, using To obtain; z subsequentevent To indicate what might happen next in a conversation, use to obtain; The construction of the reply generation module specifically includes: 3-1: The four LoRA model parameters obtained after fine-tuning are used to construct the emotional support dialogue response generation module; among them, the content set Z generated by the dialogue common sense reasoning analysis module collection It is expressed by the following formula (z): WITH collection =[of emotion ;With suggestion ;With intention ;With subsequentevent ] (With); Among them, z emotion To obtain the user's current emotions and their reasons by inputting the conversation history; suggestion Recommended information for the current user situation; intention The user's intention predicted by the model; subsequentevent For what might happen next in the conversation; The loss function Loss_response is expressed by the following formula (I): Among them, Z collection is the content set generated by the dialogue common sense reasoning analysis module; x is the target generated response; I is the total length of the response; x ＜i is the response that has been generated; C t Background prompts generated for the tth round of dialogue; s t A text description of the selected policy action; 3-2: Use the trained Model, combined with the content Z generated by the dialogue common sense reasoning analysis module collection And the selected dialogue strategy, the dialogue response x is expressed by the following formula (II) t : Among them, x t is the dialogue reply of round t; s t is a text description of the selected strategy action; Z collection The content collection generated by the dialogue common sense reasoning analysis module; C t The background prompts generated for the tth round of dialogue.

2. The emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy according to claim 1 is characterized in that: The step 1-1 selects a suitable dialogue strategy according to the current dialogue history, and converts the dialogue history together with the used dialogue strategy into a fixed code through a dialogue state encoder.

3. The emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy according to claim 1 is characterized in that: In step 1-2, after obtaining the current dialogue state encoding, a variational Bayesian inverse reinforcement learning strategy selection module is used to select the dialogue strategy. The training of the module uses a neural network to fit the posterior distribution of the reward function, and uses demonstration data for strategy selection training. At the same time, a constrained loss function is introduced to ensure consistency between the reward function fitting module and the strategy selection module.

4. The emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy according to claim 1, characterized in that: The dialogue common sense analysis and reasoning module uses the user's current emotions and their causes. emotion , Suggested information for the current user status suggestion , possible intentions in future responses intention What might happen next in the conversation? subsequentevent Four LoRA model parameters, analysis of the current conversation process and common sense generation.

5. The emotion support dialogue system based on variational Bayesian inverse reinforcement learning strategy according to claim 1, characterized in that: The response generation module combines the selected strategy and common sense information with the conversation history to generate an emotional support response.

Images