A model-based emotion-aware dialogue strategy learning method

Abstract

The present invention discloses a model-based emotion-aware dialogue strategy learning method. The method comprises the following steps: a dialogue agent performs multiple rounds of interactions with a human user to obtain real dialogue experience, performs data cleaning on the real dialogue experience according to a task scenario, and trains a dialogue strategy model; an emotion-aware world model performs parameter adjustment based on the real dialogue experience to learn the user's actions and emotional changes; and strategy learning based on simulated experience, wherein the dialogue agent interacts with the emotion-aware world model to obtain simulated dialogue experience and optimize the parameters of the dialogue strategy model. The present invention transforms emotion-aware dialogue strategy learning from a model-free learning process into a model-based learning process, thereby eliminating the original method's reliance on a user simulator, while improving the problem of poor simulation performance in the existing deep Dyna‑Q learning framework, and improving the training efficiency of the dialogue strategy model.

Description

Technical Field

[0001] The present invention relates to the field of human-computer dialogue in natural language processing, and in particular to a task-oriented dialogue strategy learning method integrating user's full emotions. Technical Background

[0002] With the breakthrough of natural language technology, human-computer dialogue systems are increasingly widely used in human-computer interaction, intelligent customer service and other fields. The rapid advancement of the country's digital economy construction has prompted various industries to improve their intelligence. In human-computer dialogue scenarios, real users directly participate in the training of models, which requires a large amount of manpower costs. Usually, low-cost user simulators are used to collect training data. However, the method of policy learning based on user simulators has some shortcomings, such as: (1) User simulators are usually built based on rules, and the deviation between their performance and real user behavior will affect the final performance of the policy model; (2) Over time, the user group of the dialogue agent may change, and the rule-based user simulator cannot observe and learn such changes; (3) User simulators need to be redesigned when migrating to tasks in different fields, which is time-consuming and labor-intensive.

[0003] To solve this problem, the deep Dyna-Q learning method can be used to model users and learn the transition probability information between environmental states, thereby transforming model-free dialogue strategy learning into a model-based learning task. Deep Dyna-Q learning uses a network called a world model to learn user behavior from real human-computer dialogue data. By allowing the dialogue agent to interact with the world model, additional simulated dialogue data is obtained and used to optimize the strategy model. Existing deep Dyna-Q learning methods do not consider the simulation of user emotions. At the same time, deep Dyna-Q learning methods are still restricted by the simulation capabilities of the world model. Current deep Dyna-Q methods usually use multi-layer perceptron networks to build world models, which makes it difficult to effectively learn the behavior patterns of users in complex dialogue scenarios, resulting in poor quality of simulated experience.

[0004] In the existing invention patent application, the conversation context of the user query is obtained, multiple candidate conversations are obtained according to the conversation context, and the candidate conversation that best matches the conversation context is selected from the multiple candidate conversations as the reply conversation. This method can only be applied to scenarios with sufficient data sets (Fu Zhenxin, Cui Shaobo, Ji Feng, Yan Rui, Zhang Ji, Chen Haiqing. Conversation method and system [P]. Cayman Islands: CN114036267A, 2022-02-11.). Summary of the invention

[0005] The model-based emotion-aware dialogue strategy learning method of the present invention combines user emotions with a model-based strategy learning method, can overcome the dependence on a user simulator in dialogue strategy learning, and improve the training efficiency of the dialogue strategy model.

[0006] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0007] A model-based emotion-aware dialogue strategy learning method includes the following steps:

[0008] S1. The dialogue agent conducts multiple rounds of interactions with human users to obtain real dialogue experience. According to the task scenario, the real dialogue experience is cleaned and the dialogue strategy model is trained.

[0009] S2, the emotion-aware world model adjusts parameters based on real conversation experience to learn the user's actions and emotional changes;

[0010] S3. Strategy learning based on simulated experience: The dialogue agent interacts with the emotion-aware world model to obtain simulated dialogue experience and optimize the parameters of the dialogue strategy model.

[0011] Furthermore, in step S1, the data cleaning method is to filter out missing, duplicated, and typing errors in the real data;

[0012] The existing Markov decision process-based framework is trained based on the real conversation experience after data cleaning, and the conversation strategy model is obtained after the training is completed.

[0013] Further, in step S2, the emotion-aware world model includes an AGRU-based conversation context modeling module, a sub-goal segmentation-based feature shaping module, a prediction layer network, and a forward prediction module;

[0014] Among them, the AGRU-based dialogue context modeling module encodes the user's goals;

[0015] The feature shaping module based on sub-goal segmentation obtains the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience according to the user goal G. t ;

[0016] The encoding of the user's goal and the sub-goal segmentation feature vector u of the tth round of interaction t and the current t-th round of dialogue state s t The prediction layer network is used as the input of the prediction layer network to predict the user action, user emotion of the current round of dialogue and the system action of the next round;

[0017] In the forward-looking prediction module, the output of the prediction layer network and the next round of dialogue system actions are used as the input of the LSTM network to predict the next round of user actions and user emotions.

[0018] Furthermore, step S2 specifically includes the following steps:

[0019] S2.1. Using the feature shaping method based on sub-goal segmentation, we obtain the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience. t , evaluate the completion progress of the t-th round of dialogue tasks in the real dialogue experience and reduce the target space; where the t-th round of dialogue tasks refers to the requests of human users when the dialogue agent interacts with human users in the t-th round;

[0020] S2.2. Conversation context modeling based on attention gated recurrent unit (AGRU) to encode user goals and capture the association between past interactions and user goals from the conversation context.

[0021] S2.3. The prediction layer network of the world model is an LSTM network. Given the dialogue state s t , dialogue context features d M , sub-target segmentation feature vector u t and the dialogue agent actions Realize the simulation of the environment by the world model;

[0022] S2.4. Forward prediction based on LSTM network. Utilizing the recursive characteristics of LSTM network, a forward prediction module is constructed to learn users’ expectations for subsequent interaction trajectories.

[0023] Furthermore, step S2.1 specifically includes the following steps:

[0024] S2.1.1. According to the dynamic segmentation algorithm, the user goal G is segmented |G| times, |G|=|P|+|Q|, and the user goal G includes request slots and constraint slots; the request slot refers to the user's request in the conversation, and the constraint slot refers to the constraint condition of the user on a certain request in the conversation; where P is the set of constraint slots and request slots that have been determined during the conversation, and Q is the set of constraint slots and request slots that have not been recognized by the conversation agent; |P| and |Q| represent the set size;

[0025] The user goal G is obtained by natural language understanding of the user's conversation. Natural language understanding uses LSTM network to predict request slots and constraint slots. See the paper "Li, X., Chen, Y.-N., Li, L., Gao, J., & Celikyilmaz, A. End-to-End Task-Completion Neural Dialogue Systems. Proceedings of the Eighth International Joint Conference on Natural Language Processing, 2017, 733–743. https: / / www.microsoft.com / en-us / research / publication / end-end- task-completion-neural-dialogue-systems / ”;

[0026] S2.1.2. Construct the sub-goal completion rate characteristic as (|P| / |G|) t , which is used to reflect the completion progress of the user's dialogue task in the tth round of interaction in the real dialogue experience, the sub-goal completion rate feature (|P| / |G|) t The mathematical meaning of is the proportion of user goals that have been completed in the tth round of dialogue. The larger the ratio, the higher the completion rate of user goals in the tth round of dialogue.

[0027] S2.1.3. Construct the sub-goal dialogue relevance feature, which is defined as the increment of the sub-goal completion rate after the tth round of interaction (|P| / |G|) t -(|P| / |G|) t-1 , which is used to reflect the contribution of the dialogue agent’s response in the tth round of interaction in the real dialogue experience to the sub-goal completion rate;

[0028] S2.1.4. Construct the feature of the average length of the sub-goal after the t-th round of interaction as |P| / (t+1). The inverse of the feature value of the average length of the sub-goal reflects the average number of interactions required for the current dialogue agent to complete a sub-goal.

[0029] S2.1.5. Directly concatenate and fuse the sub-goal completion rate feature, sub-goal dialogue relevance feature, and sub-goal average length feature into the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience. t .

[0030] Furthermore, step S2.2 specifically includes the following steps:

[0031] S2.2.1. Identify the intent, slot value, and sentiment information included in the conversation interaction in real conversation experience through natural language understanding, and obtain the vector representation of the fusion of intent, slot value, and sentiment information through the encoding network Represents the encoding of intention, slot value and sentiment information in the tth round of interaction;

[0032] S2.2.2. Extracting the user goal G after the tth round of interaction t The constraint slots and request slots involved are then encoded into a vector representation.

[0033] S2.2.3. Given all M rounds of dialogue interactions in real dialogue experience and the user goal G in the current round t t , the attention gated recurrent unit network AGRU uses the attention mechanism to encode all M rounds of dialogue interactions in the real dialogue experience in the following way:

[0034]

[0035] z t =W z tanh(W a ·attn t +b a )+b z

[0036]

[0037]

[0038]

[0039]

[0040] Among them, attn t Indicates the relevance of the t-th round of interaction to the current user goal; z t and g t is the gate control unit; t and g t is based on the relevance of the t-th round of interaction to the current user goal attn t The gating unit of the attention gated recurrent unit network AGRU built through the attention mechanism, is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction; d t is the hidden state of the attention gated recurrent unit network AGRU in the tth round of interaction, which is the hidden state d of the previous round of interaction. t-1 and the candidate state in the current t-th round of interaction In the gate control unit g t The hidden state d of the attention gated recurrent unit network AGRU in the last round of interaction is calculated under the control of t is regarded as the feature extraction result of the conversation context; W z , W a They are gate control units zt The network parameters, b a , b z is the gate control unit z t The network bias value, M is the total number of rounds of dialogue interaction, σ is the δ-function, r t is the update gate of the attention gated recurrent unit network AGRU in the tth round of interaction, W r is the r of the attention gated recurrent unit network AGRU in the tth round of interaction t The weight, W d is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction The weight of b d is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction The paranoid value.

[0041] Furthermore, step S2.3 specifically includes the following steps:

[0042] S2.3.1. Using the dialogue state s of the tth round of interaction t , dialogue context features d M and sub-target segmentation feature vector u t Initialize the initial cell state c of the LSTM network * and the hidden state h * , as follows:

[0043] c * =tanh(W c ·[s t ;d M ]+b c )

[0044] h * =tanh(W h ·u t +b h )

[0045] Among them, the initial cell state c * According to the dialogue state s of the tth round of interaction t With the dialogue context feature d M The vector obtained after splicing is calculated; W c and W h The cell states are c * and the hidden state h * The weight of b c and b h is the bias value; the dialogue state s t is the state information of the t-round dialogue, including dialogue actions and dialogue rewards; d Mis the hidden state d of the attention gated recurrent unit network AGRU obtained in step S2.2.3 based on all M rounds of dialogue interactions in real dialogue experience t , denoted as d M ; W c and W h , b c and b h Can be learned through LSTM network;

[0046] S2.3.2. Using the prediction layer network to predict the user action in the tth round of interaction User sentiment Reward t and the terminal symbol f t Make predictions; calculate the cell state c of the current t-th round of interaction through the prediction layer network t and the hidden state h t , using multi-tasking to convert the cell state c t The output is sent to a specific Linear network to predict different targets. The specific formula is:

[0047]

[0048]

[0049]

[0050] r t =Linear r (c t )

[0051] f t =Sigmoid(Linear f (c t ))

[0052] Among them, LSTM is the prediction layer network. is the action of the dialogue agent in the tth round of interaction, Linear u 、Linear e 、Linear r 、Linear f Respectively represent r t 、f t Linear layer network; Softmax and Sigmoid are Softmax activation function and Sigmoid activation function respectively.

[0053] Furthermore, step S2.4 specifically includes the following steps:

[0054] S2.4.1. Based on the hidden state h in the current t-th round of interaction t Predicting the next action in the interaction Calculated as:

[0055]

[0056] S2.4.1. Using LSTM network to predict user actions in the next round of interaction and user emotional state The forward-looking forecast is as follows:

[0057]

[0058]

[0059]

[0060] in, and The calculation of all depends on the cell state c in the tth round of interaction t and the hidden state h t , and the gradient can propagate along the LSTM network to the prediction layer network of the world model.

[0061] Furthermore, in step S3, the dialogue agent interacts with the world model to obtain simulated dialogue experience, and uses the obtained experience to optimize the parameters of the dialogue strategy model. The world model minimizes the multi-task loss Conduct training as follows:

[0062]

[0063] in, and r t are the predicted value and true value of the reward for the tth round of interaction respectively; and f are the predicted and true values ​​of the end mark of the tth round of interaction, respectively; and are the predicted value and true value of user sentiment in the tth round of interaction respectively; and are the predicted value and true value of the dialogue agent action in the t+1th round of interaction respectively; and are the predicted value and true value of the user action in the t+1th round of interaction, respectively; α is the weight coefficient of the forward prediction module loss; the MSE(·) function represents the mean square error loss, and the CE(·) function represents the cross entropy loss;

[0064] The parameters of the dialogue policy model will be based on the loss function Optimize and continuously optimize with the gradient descent algorithm To achieve convergence, the model parameters will be continuously updated during this process; take the converged The corresponding model parameters are final parameters.

[0065] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0066] 1. The present invention designs an emotion-aware deep Dyna-Q learning method to overcome the problem that the existing deep Dyna-Q model does not consider the user emotion simulation, and to improve the shortcoming of the low quality of simulation experience faced by the current deep Dyna-Q learning.

[0067] 2. Since the user's speech in task-based dialogues usually revolves around their goals, the present invention introduces user goal features in the state modeling process of the world model, which helps the world model better understand the dialogue process and improve its ability to simulate user dialogue actions and emotional changes.

[0068] 3. Psychological research shows that the psychological model in the human brain can predict the perception of the future based on the current situation, and these predictions will react to the feelings and decisions at the current moment. Therefore, the present invention introduces a forward-looking prediction module to learn the predictions of real humans for the upcoming interaction trajectory, and better model the user through this prediction information. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of the basic process of a model-based emotion-aware dialogue strategy learning method of the present invention.

[0070] Figure 2 It is a basic framework diagram of a world model of emotion perception in a model-based emotion perception dialogue strategy learning method of the present invention.

[0071] Figure 3 A schematic diagram of the basic structure of the attention gated recurrent unit network, which is the basic framework of the world model of emotion perception in the present invention. DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described below in conjunction with the accompanying drawings in the embodiments of the present invention. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0073] Embodiment 1:

[0074] A model-based emotion-aware dialogue strategy learning method, such as Figure 1 , Figure 2 and Figure 3 As shown, the following steps are included:

[0075] S1. The dialogue agent conducts multiple rounds of interactions with human users to obtain real dialogue experience. According to the task scenario, the real dialogue experience is cleaned and the dialogue strategy model is trained.

[0076] The method of data cleaning is to filter out missing, duplicate, and typographical errors in real data;

[0077] The existing Markov decision process-based framework is trained based on the real conversation experience after data cleaning, and the conversation strategy model is obtained after the training is completed.

[0078] (Reference article E.Levin, R.Pieraccini and W.Eckert, "Learning dialogue strategies within the Markov decision process framework," 1997IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, 1997, pp.72-79, doi:10.1109 / ASRU.1997.658989.)

[0079] S2, the emotion-aware world model adjusts parameters based on real conversation experience to learn the user's actions and emotional changes;

[0080] The emotion-aware world model includes an AGRU-based dialogue context modeling module, a sub-goal segmentation-based feature shaping module, a prediction layer network, and a forward prediction module;

[0081] Among them, the AGRU-based dialogue context modeling module encodes the user's goals;

[0082] The feature shaping module based on sub-goal segmentation obtains the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience according to the user goal G. t ;

[0083] The encoding of the user's goal and the sub-goal segmentation feature vector u of the tth round of interaction t and the current t-th round of dialogue state s t Together they serve as the input of the prediction layer network, which predicts the user actions and emotions of the current round of dialogue and the system actions of the next round.

[0084] In the forward-looking prediction module, the output of the prediction layer network and the next round of dialogue system actions are used as the input of the LSTM network to predict the next round of user actions and user emotions.

[0085] Step S2 specifically includes the following steps:

[0086] S2.1. Using the feature shaping method based on sub-goal segmentation, we obtain the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience. t , evaluate the completion progress of the t-th round of dialogue tasks in the real dialogue experience and reduce the target space; wherein the t-th round of dialogue tasks refers to the request of the human user when the dialogue agent interacts with the human user in the t-th round, specifically including the following steps:

[0087] S2.1.1. According to the dynamic segmentation algorithm, the user goal G is segmented |G| times, |G|=|P|+|Q|, and the user goal G includes request slots and constraint slots. Request slots refer to the user's request in the conversation, and constraint slots refer to the constraints of the user on a request in the conversation; where P is the set of constraint slots and request slots that have been determined during the conversation, and Q is the set of constraint slots and request slots that have not been recognized by the conversation agent; |P| and |Q| represent the size of the set;

[0088] The user goal G is obtained by natural language understanding of the user's conversation. Natural language understanding uses LSTM network to predict request slots and constraint slots. See the paper "Li, X., Chen, Y.-N., Li, L., Gao, J., & Celikyilmaz, A. End-to-End Task-Completion Neural Dialogue Systems. Proceedings of the Eighth International Joint Conference on Natural Language Processing, 2017, 733–743. https: / / www.microsoft.com / en-us / research / publication / end-end- task-completion-neural-dialogue-systems / ”;

[0089] S2.1.2. Construct the sub-goal completion rate characteristic as (|P| / |G|) t , which is used to reflect the completion progress of the user's dialogue task in the tth round of interaction in the real dialogue experience, the sub-goal completion rate feature (|P| / |G|) t The mathematical meaning of is the proportion of user goals that have been completed in the tth round of dialogue. The larger the ratio, the higher the completion rate of user goals in the tth round of dialogue.

[0090] S2.1.3. Construct the sub-goal dialogue relevance feature, which is defined as the increment of the sub-goal completion rate after the tth round of interaction (|P| / |G|) t -(|P| / |G|) t-1 , which is used to reflect the contribution of the dialogue agent’s response in the tth round of interaction in the real dialogue experience to the sub-goal completion rate;

[0091] S2.1.4. Construct the feature of the average length of the sub-goal after the t-th round of interaction as |P| / (t+1). The inverse of the feature value of the average length of the sub-goal reflects the average number of interactions required for the current dialogue agent to complete a sub-goal.

[0092] S2.1.5. Directly concatenate and fuse the sub-goal completion rate feature, sub-goal dialogue relevance feature, and sub-goal average length feature into the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience. t .

[0093] S2.2. Conversation context modeling is performed based on the attention gated recurrent unit (AGRU), the user goal is encoded, and the association between past interactions and user goals is captured from the conversation context. Specifically, the following steps are included:

[0094] S2.2.1. Identify the intent, slot value, and sentiment information included in the conversation interaction in real conversation experience through natural language understanding, and obtain the vector representation of the fusion of intent, slot value, and sentiment information through the encoding network Represents the encoding of intention, slot value and sentiment information in the tth round of interaction;

[0095] S2.2.2. Extracting the user goal G after the tth round of interaction t The constraint slots and request slots involved are then encoded into a vector representation.

[0096] S2.2.3. Given all M rounds of dialogue interactions in real dialogue experience and the user goal G in the current round t t , the attention gated recurrent unit network AGRU uses the attention mechanism to encode all M rounds of dialogue interactions in the real dialogue experience in the following way:

[0097]

[0098] z t =W z tanh(W a ·attn t +b a )+b z

[0099]

[0100]

[0101]

[0102]

[0103] Among them, attn t Indicates the relevance of the t-th round of interaction to the current user goal; z t and g t is the gate control unit; t and g t is based on the relevance of the t-th round of interaction to the current user goal attn t The gating unit of the attention gated recurrent unit network AGRU built through the attention mechanism, is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction; d t is the hidden state of the attention gated recurrent unit network AGRU in the tth round of interaction, which is the hidden state d of the previous round of interaction. t-1 and the candidate state in the current t-th round of interaction In the gate control unit g t The hidden state d of the attention gated recurrent unit network AGRU in the last round of interaction is calculated under the control of t is regarded as the feature extraction result of the conversation context; W z , W a They are gate control units z t The network parameters, b a 、b z is the gate control unit z t The network bias value, M is the total number of rounds of dialogue interaction, σ is the δ-function, r t is the update gate of the attention gated recurrent unit network AGRU in the tth round of interaction, W r is the r of the attention gated recurrent unit network AGRU in the tth round of interaction t The weight, W d is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction The weight of b d is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction The paranoid value.

[0104] S2.3. The prediction layer network of the world model is an LSTM network. Given the dialogue state s t , dialogue context features d M , sub-target segmentation feature vector u t and the dialogue agent actions To achieve the world model's simulation of the environment, the following steps are included:

[0105] S2.3.1. Using the dialogue state s of the tth round of interaction t , dialogue context features d M and sub-target segmentation feature vector u tInitialize the initial cell state c of the LSTM network * and the hidden state h * , as follows:

[0106] c * =tanh(W c ·[s t ;d M ]+b c )

[0107] h * =tanh(W h ·u t +b h )

[0108] Among them, the initial cell state c * According to the dialogue state s of the tth round of interaction t With the dialogue context feature d M The vector obtained after splicing is calculated; W c and W h The cell states are c * and hidden state j * The weight of b c and b h is the bias value; the dialogue state s t is the state information of the t-round dialogue, including dialogue actions and dialogue rewards; d M is the hidden state d of the attention gated recurrent unit network AGRU obtained in step S2.2.3 based on all M rounds of dialogue interactions in real dialogue experience t , denoted as d M ; W c and W h , b c and b h Can be learned through LSTM network;

[0109] S2.3.2. Using the prediction layer network to predict the user action in the tth round of interaction User sentiment Reward t and the terminal symbol f t Make predictions; calculate the cell state c of the current t-th round of interaction through the prediction layer network t and the hidden state h t , using multi-tasking to convert the cell state c t The output is sent to a specific Linear network to predict different targets. The specific formula is:

[0110]

[0111]

[0112]

[0113] r t =Linear r (c t )

[0114] f t =Sigmoid(Linear f (c t ))

[0115] Among them, LSTM is the prediction layer network. is the action of the dialogue agent in the tth round of interaction, Linear u 、Linear e 、Linear r 、Linear f Respectively represent r t 、f t Linear layer network; Softmax and Sigmoid are Softmax activation function and Sigmoid activation function respectively.

[0116] S2.4. Forward prediction based on LSTM network. Using the recursive characteristics of LSTM network, a forward prediction module is constructed to learn the user's expectations for subsequent interaction trajectories. Specifically, the following steps are included:

[0117] S2.4.1. Based on the hidden state h in the current t-th round of interaction t Predicting the next action in the interaction Calculated as:

[0118]

[0119] S2.4.2. Using LSTM network to predict user actions in the next round of interaction and user emotional state The forward-looking forecast is as follows:

[0120]

[0121]

[0122]

[0123] in, and The calculation of all depends on the cell state c in the tth round of interaction t and the hidden state h t, and the gradient can propagate along the LSTM network to the prediction layer network of the world model.

[0124] S3, strategy learning based on simulated experience, through which the dialogue agent interacts with the emotion-aware world model to obtain simulated dialogue experience and optimize the parameters of the dialogue strategy model;

[0125] The dialogue agent interacts with the world model to obtain simulated dialogue experience and uses the experience to optimize the parameters of the dialogue policy model. The world model minimizes the multi-task loss. Conduct training as follows:

[0126]

[0127] in, and r t are the predicted value and true value of the reward for the tth round of interaction respectively; and f are the predicted and true values ​​of the end mark of the tth round of interaction, respectively; and are the predicted value and true value of user sentiment in the tth round of interaction respectively; and are the predicted value and true value of the dialogue agent action in the t+1th round of interaction respectively; and are the predicted value and true value of the user action in the t+1th round of interaction, respectively; α is the weight coefficient of the forward prediction module loss; the MSE(·) function represents the mean square error loss, and the CE(·) function represents the cross entropy loss;

[0128] The parameters of the dialogue policy model will be based on the loss function Optimize and continuously optimize with the gradient descent algorithm To achieve convergence, the model parameters will be continuously updated during this process; take the converged The corresponding model parameters are final parameters.

[0129] Embodiment 2:

[0130] The difference between this embodiment and embodiment 1 is that:

[0131] In step S2.1.1, according to the dynamic segmentation algorithm, the user goal G is segmented a limited number of times, the number of times is set to 10, the user goal G includes request slots and constraint slots, the request slot refers to the user's request in the conversation, and the constraint slot refers to the constraint condition of the user on a certain request in the conversation; construct a set P that stores the constraints and request slots that have been determined during the conversation, and a set Q that stores the constraints and requests that have not yet been recognized by the conversation agent; |P| and |Q| represent the size of the set;

[0132] In step S2.1.5, the sub-goal completion rate feature, sub-goal dialogue relevance feature and sub-goal average length feature are accumulated and fused into the sub-goal segmentation feature vector u t .

[0133] In step S2.3.1, the dialogue state s of the tth round of interaction is used t , dialogue context features d M and sub-target segmentation feature vector u t Initialize the initial cell state c of the LSTM network * and the hidden state h * , as follows:

[0134] c * =softmax(W c ·[s t ;d M ]+b c )

[0135] j * =softmax(W h ·u t +b h )

[0136] Among them, the initial cell state c * According to the dialogue state s of the tth round of interaction t With the dialogue context feature d M The vector obtained after splicing is calculated; W c and W h The cell states are c * and hidden state j * The weight of b c and b h is the bias value; the dialogue state s t is the state information of the t-round dialogue, including dialogue actions and dialogue rewards; d M is the hidden state d of the attention gated recurrent unit network AGRU obtained in step S2.2.3 based on all M rounds of dialogue interactions in real dialogue experience t , denoted as d M ; W c and W h , b c and b h Can be learned through LSTM network;

[0137] In step S2.4.2, the GRU network is used to predict the user actions in the next round of interaction. and user emotional state The forward-looking forecast is as follows:

[0138]

[0139]

[0140]

[0141] in, and The calculation of all depends on the cell state c in the tth round of interaction t and the hidden state h t , and the gradient can propagate along the LSTM network to the prediction layer network of the world model.

[0142] Embodiment 3:

[0143] The difference between this embodiment and embodiment 1 is that:

[0144] In step S2.1.1, according to the dynamic segmentation algorithm, the user goal G is segmented a limited number of times, the number of times is set to 5, the user goal G includes request slots and constraint slots, the request slot refers to the user's request in the conversation, and the constraint slot refers to the constraint condition of the user on a certain request in the conversation; construct a set P that stores the constraints and request slots that have been determined during the conversation, and a set Q that stores the constraints and requests that have not yet been recognized by the conversation agent; |P| and |Q| represent the size of the set;

[0145] In step S2.3.1, the dialogue state s of the tth round of interaction is used t , dialogue context features d M and sub-target segmentation feature vector u t Initialize the initial cell state c of the LSTM network * and the hidden state h * , as follows:

[0146] c * =softmax(W c ·[s t ;d M ]+b c )

[0147] h * =softmax(W h ·u t +b h )

[0148] Among them, the initial cell state c * According to the dialogue state s of the tth round of interaction t With the dialogue context feature d M The vector obtained after splicing is calculated; W c and W h The cell states are c* and the hidden state h * The weight of b c and b h is the bias value; the dialogue state s t is the state information of the t-round dialogue, including dialogue actions and dialogue rewards; d M is the hidden state d of the attention gated recurrent unit network AGRU obtained in step S2.2.3 based on all M rounds of dialogue interactions in real dialogue experience t , denoted as d M ; W c and W h , b c and b h Can be learned through LSTM network;

[0149] In step S2.4.2, the GRU network is used to predict the user actions in the next round of interaction. and user emotional state The forward-looking forecast is as follows:

[0150]

[0151]

[0152]

[0153] in, and The calculation of all depends on the cell state c in the tth round of interaction t and the hidden state h t , and the gradient can propagate along the LSTM network to the prediction layer network of the world model.

[0154] In step S3, based on the simulated experience strategy learning, the dialogue agent interacts with the world model to obtain simulated dialogue experience and optimize the parameters of the dialogue strategy model;

[0155] The dialogue agent interacts with the world model to obtain simulated dialogue experience and uses the experience to optimize the parameters of the dialogue policy model. The world model minimizes the multi-task loss. Conduct training as follows:

[0156]

[0157] in, and r t are the predicted value and true value of the reward at the moment of the tth round of interaction; and f are the predicted value and true value of the dialogue end marker at the t-th dialogue round, respectively; and are the predicted value and true value of user sentiment in the tth round of interaction respectively; and are the predicted value and true value of the dialogue agent action in the t+1th round of interaction respectively; and are the predicted value and true value of the user action in the t+1th round of interaction, respectively; α is the weight coefficient of the forward prediction module loss, which is 0.8; the MSE(·) function represents the mean square error loss, and the CE(·) function represents the cross entropy loss.

[0158] The parameters of the dialogue policy model will be based on the loss function Optimize and continuously optimize with the gradient descent algorithm To achieve convergence, the model parameters will be continuously updated during this process. The corresponding model parameters are final parameters.

[0159] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that a variety of equivalent changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A model-based emotion-aware dialogue strategy learning method, characterized in that: The following steps are involved: S1. The dialogue agent conducts multiple rounds of interactions with human users to obtain real dialogue experience. According to the task scenario, the real dialogue experience is cleaned and the dialogue strategy model is trained. S2, the emotion-aware world model adjusts parameters based on real conversation experience to learn the user's actions and emotional changes; The emotion-aware world model includes a conversation context modeling module based on the attention gated recurrent unit network AGRU, a feature shaping module based on sub-goal segmentation, a prediction layer network, and a forward prediction module; Among them, the dialogue context modeling module based on the attention gated recurrent unit network AGRU encodes the user's goal; The feature shaping module based on sub-goal segmentation obtains the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience according to the user goal G. t ; Encoding of user goals and sub-goal segmentation feature vector u of the tth round of interaction t and the current t-th round of dialogue state s t The prediction layer network is used as the input of the prediction layer network to predict the user action, user emotion of the current round of dialogue and the system action of the next round; In the forward-looking prediction module, the output of the prediction layer network and the next round of dialogue system actions are used as the input of the LSTM network to predict the next round of user actions and user emotions; S3. Strategy learning based on simulated experience: The dialogue agent interacts with the emotion-aware world model to obtain simulated dialogue experience and optimize the parameters of the dialogue strategy model.

2. The method for learning a model-based emotion-aware dialogue strategy according to claim 1, characterized in that: In step S1, the data cleaning method is to filter out missing, duplicated, and typing errors in the real data; The existing Markov decision process-based framework is trained based on the real conversation experience after data cleaning, and the conversation strategy model is obtained after the training is completed.

3. The method for learning a model-based emotion-aware dialogue strategy according to claim 1, characterized in that: Step S2 specifically includes the following steps: S2.

1. Using the feature shaping method based on sub-goal segmentation, we obtain the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience. t , evaluate the completion progress of the t-th round of dialogue tasks in the real dialogue experience and reduce the target space; where the t-th round of dialogue tasks refers to the requests of human users when the dialogue agent interacts with human users in the t-th round; S2.

2. We model the conversation context based on the attention gated recurrent unit network AGRU, encode the user goal, and capture the association between past interactions and user goals from the conversation context. S2.

3. The prediction layer network of the world model is an LSTM network. Given the dialogue state s t , dialogue context features d M , sub-target segmentation feature vector u t and the dialogue agent actions Realize the simulation of the environment by the world model; S2.

4. Forward prediction based on LSTM network. Utilizing the recursive characteristics of LSTM network, a forward prediction module is constructed to learn users’ expectations for subsequent interaction trajectories.

4. The method for learning a model-based emotion-aware dialogue strategy according to claim 3, characterized in that: Step S2.1 specifically includes the following steps: S2.1.

1. According to the dynamic segmentation algorithm, the user goal G is segmented |G| times, |G|=|P|+|Q|, and the user goal G includes request slots and constraint slots; the request slot refers to the user's request in the conversation, and the constraint slot refers to the constraint condition of the user on a certain request in the conversation; where P is the set of constraint slots and request slots that have been determined during the conversation, and Q is the set of constraint slots and request slots that have not been recognized by the conversation agent; |P| and |Q| represent the set size; S2.1.

2. Construct the sub-goal completion rate characteristic as (|P| / |G|) t , which is used to reflect the completion progress of the user's dialogue task in the tth round of interaction in the real dialogue experience, the sub-goal completion rate feature (|P| / |G|) t The mathematical meaning of is the proportion of user goals that have been completed in the tth round of dialogue. The larger the ratio, the higher the completion rate of user goals in the tth round of dialogue. S2.1.

3. Construct the sub-goal dialogue relevance feature, which is defined as the increment of the sub-goal completion rate after the tth round of interaction (|P| / |G|) t -(|P| / |G|) t-1 , which is used to reflect the contribution of the dialogue agent’s response in the tth round of interaction in the real dialogue experience to the sub-goal completion rate; S2.1.

4. Construct the feature of the average length of the sub-goal after the t-th round of interaction as |P| / (t+1). The inverse of the feature value of the average length of the sub-goal reflects the average number of interactions required for the current dialogue agent to complete a sub-goal. S2.1.

5. Directly concatenate and fuse the sub-goal completion rate feature, sub-goal dialogue relevance feature, and sub-goal average length feature into the sub-goal segmentation feature vector u of the t-th round of interaction in the real dialogue experience. t .

5. The method for learning a model-based emotion-aware dialogue strategy according to claim 3, characterized in that: Step S2.2 specifically includes the following steps: S2.2.

1. Identify the intention, slot value and sentiment information included in the dialogue interaction in the real dialogue experience through natural language understanding, and obtain the vector of the fusion of intention, slot value and sentiment information through the encoding network. Represents the encoding of intention, slot value and sentiment information in the tth round of interaction; S2.2.

2. Extracting the user goal G after the tth round of interaction t The constraint slots and request slots involved are then encoded to obtain the vector S2.2.

3. Given all M rounds of dialogue interactions in real dialogue experience and the user goal G in the current round t t , the attention gated recurrent unit network AGRU uses the attention mechanism to encode all M rounds of dialogue interactions in the real dialogue experience in the following way: z t =W z tanh(W a ·attn t +b a )+b z Among them, attn t Indicates the relevance of the t-th round of interaction to the current user goal; z t and g t is based on the relevance of the t-th round of interaction to the current user goal attn t The gating unit of the attention gated recurrent unit network AGRU built through the attention mechanism, is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction; d t is the hidden state of the attention gated recurrent unit network AGRU in the tth round of interaction, which is the hidden state d of the previous round of interaction. t-1 and the candidate state in the current t-th round of interaction In the gate control unit g t The hidden state d of the attention gated recurrent unit network AGRU in the last round of interaction is calculated under the control of t is regarded as the feature extraction result of the conversation context; W z , W a They are gate control units z t The network parameters, b a 、b z is the gate control unit z t The network bias value, M is the total number of rounds of dialogue interaction, σ is the δ-function, o t is the update gate of the attention gated recurrent unit network AGRU in the tth round of interaction, W r is the o of the attention gated recurrent unit network AGRU in the tth round of interaction t The weight of b r is the O of the attention gated recurrent unit network AGRU in the tth round of interaction t The paranoid value of W d is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction The weight of b d is the candidate state of the attention gated recurrent unit network AGRU in the tth round of interaction The paranoid value.

6. The method for learning a model-based emotion-aware dialogue strategy according to claim 3, characterized in that: Step S2.3 specifically includes the following steps: S2.3.

1. Using the dialogue state s of the tth round of interaction t , dialogue context features d M and sub-target segmentation feature vector u t Initialize the initial cell state c of the LSTM network * and the hidden state h * , as follows: c * =tanh(W c ·[s t ;d M ]+b c ) h * =tanh(W h ·u t +b h ) Among them, the initial cell state c * According to the dialogue state s of the tth round of interaction t With the dialogue context feature d M The vector obtained after splicing is calculated; W c and W h The cell states are c * and the hidden state h * The weight of b c and b h is the bias value; the dialogue state s t is the state information of the t-round dialogue, including dialogue actions and dialogue rewards; d M is the hidden state d of the attention gated recurrent unit network AGRU obtained in step S2.2.3 based on all M rounds of dialogue interactions in real dialogue experience t , denoted as d M ; W c and W h 、b c and b h Learned through LSTM network; S2.3.

2. Using the prediction layer network to predict the user actions in the tth round of interaction User sentiment Reward t and the terminal symbol f t Make predictions; calculate the cell state c of the current t-th round of interaction through the prediction layer network t and the hidden state h t , using multi-tasking to convert the cell state c t The output is sent to a specific Linear network to predict different targets. The specific formula is: r t =Linear r (c t ) f t =Sigmoid(Linear f (c t )) Among them, LSTM is the prediction layer network. is the action of the dialogue agent in the tth round of interaction, Linear u 、Linear e 、Linear r 、Linear f Respectively represent r t 、f t Linear layer network; Softmax and Sigmoid are Softmax activation function and Sigmoid activation function respectively.

7. The method for learning a model-based emotion-aware dialogue strategy according to claim 3, characterized in that: Step S2.4 specifically includes the following steps: S2.4.

1. Based on the hidden state h in the current t-th round of interaction t Predicting the next action in the interaction Calculated as: S2.4.

2. Using LSTM network to predict user actions in the next round of interaction and user emotional state The forward-looking forecast is as follows: in, and The calculation of all depends on the cell state c in the tth round of interaction t and the hidden state h t , and the gradient can propagate along the LSTM network to the prediction layer network of the world model.

8. The method for learning a model-based emotion-aware dialogue strategy according to claim 1, characterized in that: In step S3, the dialogue agent interacts with the world model to obtain simulated dialogue experience, and uses the obtained experience to optimize the parameters of the dialogue policy model. The world model is trained by minimizing the multi-task loss L, as follows: in, and r t are the predicted value and true value of the reward for the tth round of interaction respectively; and f t are the predicted value and true value of the end mark of the tth round of interaction respectively; and are the predicted value and true value of user sentiment in the tth round of interaction respectively; and are the predicted value and true value of the dialogue agent action in the t+1th round of interaction respectively; and are the predicted value and true value of the user action in the t+1th round of interaction, respectively; α is the weight coefficient of the forward prediction module loss; the MSE(·) function represents the mean square error loss, and the CE(·) function represents the cross entropy loss.

9. The method for learning a model-based emotion-aware dialogue strategy according to claim 8, characterized in that: The parameters of the dialogue policy model will be based on the loss function Optimize and continuously optimize with the gradient descent algorithm To achieve convergence, the model parameters will be continuously updated during this process; take the converged The corresponding model parameters are final parameters.

Images