A method and system for empathy dialogue training based on scenario adaptation

Abstract

The present invention relates to an empathy dialogue training method and system based on scenario adaptation, belonging to the field of artificial intelligence technology. The empathy dialogue training method and system based on scenario adaptation of the present invention enables participants A and B to jointly train a dialogue generation model to generate personalized medical and travel dialogue responses; through the federated learning framework, the data privacy and security of the participants are protected, and the training efficiency is improved; the aggregation of model parameters and the distribution and update of the global model ensure the knowledge sharing and iterative optimization of the model among the participants, and improve the performance and adaptability of the dialogue generation model; at the same time, the method has a wide range of application prospects and is also applicable to dialogue generation tasks in other fields, such as customer service dialogues, intelligent assistants, etc.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an empathy dialogue training method and system based on scenario adaptation. Background Art

[0002] With the development and application of deep learning technology in the field of natural language processing, dialogue system tasks have made breakthrough progress. Dialogue systems are generally divided into task completion, chat and question-answering types. Task completion dialogues mainly help users achieve preset goals; chat dialogues mainly simulate normal human communication through multiple rounds of dialogue; question-answering dialogues generally use knowledge graphs and retrieval methods to solve the need to acquire knowledge. At present, dialogue systems mainly focus on solving content generation quality and context relevance, lacking consideration of emotional changes and personalized characteristics of both parties in the dialogue, ignoring changes in situations and emotions during the dialogue process, and the coordination of emotions and knowledge, making it impossible for the two parties in the dialogue to resonate, and even causing conflicts between emotions and knowledge, making it impossible for the dialogue to be in-depth and continuous. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] The technical problem to be solved by the present invention is: to propose an empathy dialogue training method and system based on scenario adaptation, through federated reinforcement learning, distributed training dialogue generation model, while introducing external knowledge, integrating personality background and emotional characteristics, unifying knowledge and emotion, and performing differentiated training on each client.

[0005] (II) Technical solution

[0006] In order to solve the above technical problems, the present invention provides an empathy dialogue training method based on scenario adaptation, comprising the following steps:

[0007] Step 1: Initialize the global model, which is the dialogue model;

[0008] Before federated learning begins, a global model is initialized. The transformer-based pre-trained model GPT-2 is used to train the global model using a multi-round dialogue training set. First, the input text is passed through the word embedding layer to convert each word into a corresponding word vector representation. The input sequence is X = [x1, x2, ..., x n ], where x i represents the index of the i-th word, n is the number of words in the input sequence; the word embedding layer converts the index x of each word i Mapped to a d-dimensional word vector Get the word embedding representation E = [e1, e2, ..., e n]; GPT-2 uses a multi-head self-attention mechanism to capture the contextual relationship in the input sequence. The multi-head self-attention mechanism includes three steps: ① Calculation of query, key and value. The word embedding representation E of the input sequence is transformed by three matrices Q=E·WQ, K=E·WK, V=E·WV to obtain the query vector Q, key vector K and value vector V, where WQ, WK and WV are learnable parameter matrices respectively; ② Calculation of attention score. The dot product between the query vector Q and the key vector K is calculated, and then the attention score is calculated by the scaling factor d. k Scaling to get the attention score: The softmax function is a normalized exponential function that converts the result value into a probability between [0,1]; ③ Weighted summation, weighted summation of the attention score and the value vector V, to obtain the output of self-attention: Output = Attention V; In order to retain the position information in the input sequence, gpt-2 uses position encoding to add a fixed vector representation to each position. The position encoding formula is as follows: Among them, pos represents the position index, i represents the dimension index, and d represents the dimension of the word vector; after self-attention calculation and position encoding, the input sequence is added to the output Output of self-attention through residual connection; after each attention layer, gpt-2 also contains a feedforward neural network layer for nonlinear transformation and mapping of the features of each position, and the feedforward neural network layer consists of two linear transformations and an activation function; the last layer of the gpt-2 model is a linear transformation and a softmax function, which is used to map the final output of the model to the probability distribution on the vocabulary: output_probs = softmax(output·W2+b2), where W2 and b2 are learnable parameter matrices and bias vectors, respectively;

[0009] Step 2: Distribute the global model and set the initial global model parameters θ g Send to each client {U1,U2,...,U m};

[0010] Step 3: Local model training: The server sends the global model parameters obtained in the previous iteration to the client, and the client sends the global model parameters θ sent by the server. g Initialize local model parameters θ i and the local reference model parameters θ i ′, each client uses its own local data set to train the local model. The data set is the conversation data set. The data sets of all clients are marked as {D1, D2, ..., D m}; Next, update the model parameters through the optimization algorithm: Among them, θ iis the local model parameter of the i-th client, η is the learning rate, L(θ i ,D i ) is the loss function; in local model training, the gpt-2 model is optimized based on the tuning method of reinforcement learning proximal policy optimization PPO;

[0011] Step 4: Parameter upload and aggregation. Each client uploads the locally trained and updated model parameters to the server. After receiving the parameters, the server aggregates the parameters through the aggregation algorithm to obtain the global model parameters θ g ;

[0012] Step 5: Model update: the server updates the aggregated global model parameters θ g After receiving the global model parameters, the client uses them as the local model parameters for the next round. After replacing the local model with the global model, the original local model parameters are used as the reference model parameters, i.e., θ i =θ g ,θ i ′=θ i-1 ;

[0013] Step 6: Iterate training and repeat steps 3 to 5 until the preset stopping conditions are met, such as reaching the maximum number of iterations or model convergence.

[0014] Preferably, in step 1, the gpt-2 model training also uses a cross entropy function based on maximum likelihood estimation and performs parameter optimization through a gradient descent algorithm.

[0015] Preferably, PPO is a policy gradient based optimization algorithm that attempts to calculate a new policy at each iteration to minimize the loss function while ensuring that the deviation from the policy of the previous iteration is relatively small.

[0016] Preferably, the specific process of optimizing the gpt-2 model is as follows:

[0017] (1) Initialize the local model, θ i =θ g At the same time, if it is the first time to receive the global model from the server, the local reference model parameters θ are initialized using the global model parameters i ′=θ g Otherwise, use the model parameters of the local model at the previous moment to assign θ i ′=θ i-1 ; Then the local data set samples X={x1,x2,...,x n} input into the local model, execute action a and get output Y = {y1,y2,...,y n}, where yi contains the input x i ;

[0018] (2) {y1,y2,...,y n}Input into the reward model, the sentiment analysis model is used as the reward model to determine whether it is similar to the above sentiment, and the score r is obtained i ;

[0019] (3) The generation process of the gpt-2 model is regarded as a policy network, that is, mapping the input sequence to the probability distribution of the output sequence; using the current policy network to generate a batch of sequence samples, and calculating the log-likelihood value of each input sequence; for the input sequence X, the output sequence generated by the policy network is Y, and the log-likelihood value is calculated: log_probs = logP(Y|X,θ), where θ is the parameter of the policy network; in order to use the PPO algorithm for policy optimization, the advantage estimate of each sample is calculated to measure the improvement of the current strategy over the old strategy; the advantage estimate is calculated using the value function-based method GAE, which is the weighted average of multiple value estimates, and the advantage estimate value at time t is estimated based on time t+1:

[0020]

[0021] in, is the advantage estimate at time t, is the time series difference error of the sample at time t+1, γ is the discount factor, and λ is the GAE parameter; the proximal strategy optimization penalty method is used to optimize the objective function, and the objective function is as follows:

[0022]

[0023] Among them, s t is the state, a t For action, is the sampled state s under the π strategy at time t t and action a t The expected value, p θ (a t |s t ) is the probability of taking action in state θ at time t, p θ′ (a t |s t ) is the probability of the action in the state θ′ at time t, θ is the parameter of the policy network to be optimized, and θ′ refers to the parameter sampling data of the action generated by the actual interaction with the environment, from which the state s is sampled. t 、Action a t Then, calculate its advantage estimate A θ′ (s t ,a t ), the ultimate goal is to use θ′ to adjust θ; the proximal strategy is used to optimize the penalty, as shown in the following formula:

[0024]

[0025] Among them, β is the penalty term, KL(θ,θ′) refers to the KL divergence value of θ and θ′, and β is dynamically adjusted using adaptive KL penalty; if KL(θ,θ′)<KLmax, increase β;

[0026] If KL(θ,θ′)>KLmax, reduce β, where KLmax refers to the maximum value of the KL divergence.

[0027] (4) Strategy optimization: Use the gradient descent method to optimize the parameters θ of the policy network to maximize the optimization objective function; calculate the gradient of the objective function with respect to θ Update the parameters of the policy network:

[0028]

[0029] Among them, α is the learning rate;

[0030] (5) Iterative optimization: repeat steps (3) and (4) to perform multiple rounds of strategy evaluation and optimization. In each round of iteration, by generating new samples and calculating advantage estimates, the parameters of the policy network are updated, and the generation performance of the model is gradually improved.

[0031] Preferably, after receiving the parameters, the server aggregates the parameters through a dynamic weighted aggregation algorithm to obtain the global model parameter θ g Dynamic weighted aggregation method, the formula for parameter aggregation is:

[0032]

[0033] Where m is the number of clients, θ i is the model parameter of the ith client, w i is the weight of the model parameter of the ith client.

[0034] Preferably, the conversation data set includes a medical conversation data set and a hotel booking conversation data set.

[0035] The present invention also provides a system for implementing the method.

[0036] The present invention also provides an application of the method in a customer service dialogue generation task.

[0037] The present invention also provides an application of the method in an intelligent assistant dialogue generation task.

[0038] The present invention also provides an application of the method in the field of artificial intelligence technology.

[0039] (III) Beneficial effects

[0040] The dialogue generation method based on federated reinforcement learning of the present invention enables participants A and B to jointly train a dialogue generation model to generate personalized dialogue responses in the medical and travel fields; through the federated learning framework, the data privacy and security of the participants are protected, and the training efficiency is improved; the aggregation of model parameters and the distribution and update of the global model ensure the knowledge sharing and iterative optimization of the model among the participants, and improve the performance and adaptability of the dialogue generation model; at the same time, the method has broad application prospects and is also applicable to dialogue generation tasks in other fields, such as customer service dialogues, intelligent assistants, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the empathy dialogue training method based on federated reinforcement learning of the present invention;

[0042] Figure 2 This is a schematic diagram of the empathy dialogue training method based on federated reinforcement learning of the present invention;

[0043] Figure 3 This is an architecture diagram of the empathy dialogue training method based on federated reinforcement learning of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0045] At present, mainstream dialogue systems mainly focus on solving the quality of response content generation and context relevance, and rarely consider the emotional characteristics and personalized characteristics of the two parties in the dialogue. Some dialogue generation methods treat emotions as a single static variable, ignoring the changes in emotions during the dialogue process and failing to generate resonance. Secondly, when introducing external knowledge, they do not comprehensively consider the coordination between knowledge and emotions, resulting in possible conflicts between the two. Therefore, in actual dialogues, it is difficult to generate empathy, making it impossible for the dialogue to go on in depth.

[0046] The present invention proposes an empathy dialogue training method and system based on scenario adaptation. Through federated reinforcement learning and distributed training dialogue generation model, external knowledge is introduced while integrating human background and emotional characteristics, knowledge and emotion are unified, and each client performs differentiated training. Among them, firstly, the base model uses the transformer pre-training model, which solves the problem that the Seq2Seq model encodes the input sequence into a fixed-length semantic feature, resulting in partial information loss. The encoding layer of the Seq2Seq model only uses the same vector to calculate the hidden state, while the transformer pre-training model uses Attention to input different semantic features at each time of the decoding part, which can solve the long-term dependency problem caused by information loss during the calculation process and ensure parallel calculation. Then, the distributed optimization model of the federated learning framework is adopted to reduce the training cost, and the transformer-based pre-training model is sent from the server to each client. The client is trained according to the model parameters and the local environment. Each client is in a different environment, so the trained model has personalized characteristics. When the global model is sent to each client, the local model previously trained by the client will not be directly abandoned. In contrast, the global model is regarded as an initial model or reference model, and the client performs further local model training based on it. After the model training is completed, each client uploads the updated parameters to the server to fuse the global model and the local model. The goal of the fusion is to combine the global knowledge of the global model with the personalized knowledge of the local model to improve the performance and adaptability of the model. The global model will be updated and aggregated according to the model parameters uploaded by the client in each iteration to reflect the contribution of the client. For the client model, in order to better incorporate emotional features, reinforcement learning is used to update its strategy, evaluate the model output and feedback, and update the model parameters through feedback. This joint training method enables clients to jointly build and update a more representative and generalized global model while protecting data privacy.

[0047] In this invention, the server-side model is called the global model, and each client model is called a local model. The global model aggregates the model parameters or gradients uploaded by the client through the federated learning framework and updates them. The updated model is then sent to the client as the initial model or reference model, based on which further local model training is performed. Figures 1 to 3 The method of the present invention generally proceeds as follows:

[0048] 1. Initialize the global model. First, build a transformer-based pre-training model. Here, we use the GPT-2 model. Then, use the constructed multi-round dialogue training dataset to train the model to form an initial model that can realize simple dialogue.

[0049] 2. Distribute the global model. The server sends the initial model to the client. After receiving the model, the client inputs the conversation data under different environments and uses the model to generate conversation replies.

[0050] 3. Local model training. Each client uses the local data set to train the received global model. The reward model is used to capture the sentiment of the above context, and sentiment analysis is performed on the generated dialogue responses to compare the sentiment consistency of the two.

[0051] 4. Model parameter update. Optimize the model through the policy optimization algorithm PPO;

[0052] 5. Upload parameter model. The client uploads the trained model to the server. In the present invention, only the difference or gradient information of the parameters is uploaded to reduce communication overhead and protect data privacy.

[0053] 6. Aggregate model parameters. The server updates the global model by aggregating the models uploaded by each client.

[0054] 7. Iterative update: After the server updates the model, it sends the updated model parameters to the client and continues iterative update.

[0055] The detailed design process of the present invention is as follows:

[0056] Step 1: Initialize the global model, which is the dialogue model;

[0057] Before federated learning begins, a global model needs to be initialized. In this paper, the transformer-based pre-trained model GPT-2 is used to train the global model using a multi-round dialogue training set. First, the input text is passed through the word embedding layer to convert each word into a corresponding word vector representation. The input sequence is X = [x1, x2, ..., xn], where x i represents the index of the i-th word, n is the number of words in the input sequence; the word embedding layer converts the index x of each word i Mapped to a d-dimensional word vector Get the word embedding representation E = [e1, e2, ..., e n]; GPT-2 uses a multi-head self-attention mechanism to capture the contextual relationship in the input sequence. The multi-head self-attention mechanism includes three key steps: ① Calculation of query, key and value. The word embedding representation E of the input sequence is transformed by three matrices Q=E·WQ, K=E·WK, V=E·WV to obtain the query vector Q, key vector K and value vector V, where WQ, WK and WV are learnable parameter matrices respectively; ② Calculation of attention score. The dot product between the query vector Q and the key vector K is calculated, and then the scale factor d is used to calculate the attention score. k Scaling to get the attention score: Softmax() is a normalized exponential function that converts the result value into a probability between [0,1]; ③ Weighted summation, weighted summation of the attention score and the value vector V, to obtain the output of self-attention: Output = Attention V. In order to retain the position information in the input sequence, gpt-2 uses position encoding to add a fixed vector representation to each position. The position encoding formula is as follows: Among them, pos represents the position index, i represents the dimension index, and d represents the dimension of the word (embedding) vector; after the self-attention calculation and position encoding, the input sequence is added to the output Output of the self-attention through the residual connection to avoid the gradient vanishing problem. After each attention layer (the entire layer that calculates attention is called the attention layer, and the encoder in GPT2 can be generally divided into the attention layer and the feedforward neural network layer), GPT-2 also contains a feedforward neural network layer for nonlinear transformation and mapping of the features of each position. The feedforward neural network layer consists of two linear transformations and an activation function. The last layer of the GPT-2 model is a linear transformation and a softmax function, which is used to map the final output of the model to the probability distribution on the vocabulary: output_probs = softmax (output W2 + b2), where W2 and b2 are the learnable parameter matrix and bias vector, respectively. Here, the final result is linearly transformed. GPT-2 model training also uses a cross entropy function based on maximum likelihood estimation to optimize parameters through a gradient descent algorithm.

[0058] Step 2: Distribute the global model and set the initial global model parameters θ g Send to each client {U1,U2,...,U m}.

[0059] Step 3: Local model training: The server sends the global model parameters obtained in the previous iteration to the client, and the client sends the global model parameters θ sent by the server. g Initialize local model parameters θ i and the local reference model parameters θ i′, each client uses its own local data set to train the local model. The data set is the conversation data set. The data sets of all clients are marked as {D1, D2, ..., D m}; Next, update the model parameters through the optimization algorithm: Among them, θ i is the local model parameter of the i-th client, η is the learning rate, L(θ i ,D i ) is the loss function. In local model training, the present invention optimizes the gpt-2 model based on the tuning method of reinforcement learning proximal policy optimization PPO (Proximal Policy Optimization). PPO is an optimization algorithm based on policy gradient, which tries to calculate a new strategy in each iteration, so that the loss function can be minimized while ensuring that the deviation from the strategy of the previous iteration is relatively small. It mainly includes key steps such as updating the policy network, policy evaluation, calculating advantage evaluation and optimizing the target. The specific process of optimizing the gpt-2 model is as follows:

[0060] (1) Initialize the local model, θ i =θ g At the same time, if it is the first time to receive the global model from the server, the local reference model parameters θ are initialized using the global model parameters i ′=θ g Otherwise, use the model parameters of the local model at the previous moment to assign θ i ′=θ i-1 Then the local dataset samples X={x1,x2,...,x n} input into the local model, execute action a and get output Y = {y1,y2,...,y n}, where yi contains the input x i .

[0061] (2) {y1,y2,...,y n}Input the reward model, the reward model can use a classification model or a manual judgment method. In the present invention, the sentiment analysis model is used as the reward model to determine whether it is similar to the above sentiment and obtain the score r i .

[0062] (3) The generation process of the gpt-2 model is regarded as a policy network, that is, mapping the input sequence to the probability distribution of the output sequence. Use the current policy network to generate a batch of sequence samples and calculate the log-likelihood value of each input sequence. For the input sequence X, the output sequence generated by the policy network is Y, and the log-likelihood value is calculated: log_probs = logP(Y|X,θ), where θ is the parameter of the policy network. In order to use the PPO algorithm for policy optimization, it is necessary to calculate the advantage estimate of each sample to measure the improvement of the current strategy over the old strategy. The advantage estimate is calculated using the value function-based method GAE (generalized advantage estimation). GAE is the weighted average of multiple value estimates. In order to quickly estimate the estimated values ​​at all moments in the sequence, the moment t is estimated based on the moment t+1:

[0063]

[0064] in, is the advantage estimate at time t, is the time series difference error of the sample at time t+1, γ is the discount factor, and λ is the GAE parameter. The core of model optimization using the PPO algorithm is to maximize the objective function while controlling the amplitude of the policy update to maintain the stability of the policy. The objective function can be optimized using the proximal policy optimization penalty (PPO-penalty) or proximal policy optimization clipping (PPO-clip). In the present invention, the proximal policy optimization penalty method is used to optimize the objective function. The objective function is as follows:

[0065]

[0066] Among them, s t is the state, a t For action, is the sampled state s under the π strategy at time t t and action a t The expected value, p θ (a t |s t ) is the probability of taking action in state θ at time t, p θ′ (a t |s t ) is the probability of the action in the state θ′ at time t, θ is the parameter of the policy network to be optimized, and θ′ refers to the parameter sampling data of the action generated by the actual interaction with the environment, from which the state s is sampled. t 、Action a t Then, calculate its advantage estimate A θ′ (s t ,a t), the ultimate goal is to use θ′ to adjust θ. The proximal strategy optimization penalty is used for optimization, as shown in the following formula.

[0067]

[0068] Among them, β is the penalty term, KL(θ,θ′) refers to the KL divergence value of θ and θ′, and β is dynamically adjusted using adaptive KL penalty. If KL(θ,θ′)<KLmax, increase β; if KL(θ,θ′)>KLmax, reduce β, and KLmax refers to the maximum value of KL divergence.

[0069] (4) Strategy optimization: Use the gradient descent method to optimize the parameters θ of the policy network to maximize the optimization objective function. Update the parameters of the policy network:

[0070]

[0071] Here, α is the learning rate.

[0072] (5) Iterative optimization: repeat steps (3) and (4) to perform multiple rounds of strategy evaluation and optimization. In each round of iteration, new samples are generated and advantage estimates are calculated.

[0073] Update the parameters of the policy network and gradually improve the generation performance of the model.

[0074] Step 4: Parameter upload and aggregation. Each client uploads the locally trained and updated model parameters to the server. After receiving the parameters, the server aggregates the parameters through the aggregation algorithm to obtain the global model parameters θ g The present invention proposes a dynamic weighted aggregation method. The formula for parameter aggregation is:

[0075]

[0076] Where m is the number of clients, θ i is the model parameter of the ith client, w i is the weight of the model parameter of the ith client.

[0077] Step 5: Model update: the server updates the aggregated global model parameters θ g After receiving the global model parameters, the client uses them as the local model parameters for the next round. In order to prevent the local model from losing personalized features, after replacing the local model with the global model, the original local model parameters are used as the reference model parameters, that is, θ i =θ g ,θ i ′=θ i-1 .

[0078] Step 6: Iterate training and repeat steps 3 to 5 until the preset stopping conditions are met, such as reaching the maximum number of iterations or model convergence.

[0079] The following is a specific embodiment, showing the specific steps and process of the empathy dialogue training method based on federated reinforcement learning:

[0080] 1. Client selection

[0081] 1) Client A: holds a medical conversation dataset that represents conversation scenarios in the medical field.

[0082] 2) Client B: holds a hotel booking conversation dataset, which represents conversation scenarios in the travel field.

[0083] 2. Construction of Federated Learning Framework

[0084] Initialize the global reinforcement learning model, including the model structure and parameters for dialogue generation.

[0085] 3. Training for Reinforcement Learning Dialogue Generation

[0086] 1) Client A performs local training:

[0087] Use the global model as the initial model and perform reinforcement learning training on the local medical dialogue dataset.

[0088] Conduct conversational interactions with users in the medical field to collect conversation data and reward signals.

[0089] Based on local data and reward signals, the update of model parameters is calculated, and the dialogue generation model is optimized through gradient descent method.

[0090] 2) Client B performs local training:

[0091] Use the global model as the initial model and perform reinforcement learning training on the local hotel booking dialogue dataset.

[0092] Conduct conversational interactions with users in the medical field to collect conversation data and reward signals.

[0093] Based on local data and reward signals, the update of model parameters is calculated, and the dialogue generation model is optimized through gradient descent method.

[0094] 4. Model parameter aggregation

[0095] 1) Client A uploads the updated model parameters to the server.

[0096] 2) Client B uploads the updated model parameters to the server.

[0097] 3) The server updates the model parameters uploaded by the client, aggregates the parameters using a dynamic weighting method, and generates updated global model parameters.

[0098] 5. Global model distribution and update

[0099] 1) The server model sends the updated global model parameters to client A.

[0100] 2) The server model sends the updated global model parameters to client B

[0101] 6. Iterative training

[0102] 1) After receiving the global model parameters, client A continues training on the local medical dialogue dataset and generates personalized medical dialogue responses.

[0103] 2) After receiving the global model parameters, client B continues training on the local hotel booking dialogue dataset and generates personalized medical dialogue responses.

[0104] 7. Dialogue Generation and Evaluation

[0105] 1) Client A uses the locally trained model to interact with users in the medical field and generate personalized medical dialogue responses.

[0106] 2) Client B uses the locally trained model to interact with users in the travel field and generate personalized travel dialogue responses.

[0107] 3) Dialogue quality evaluation: The dialogue responses generated by client A and client B are evaluated based on evaluation indicators such as dialogue quality and sentiment tendency.

[0108] 8. Iterative training and updating

[0109] 1) Clients A and B optimize the local model based on the results of the conversation quality assessment, calculate the update of the model parameters, and upload them to the server.

[0110] 2) The server updates the model parameters uploaded by clients A and B, aggregates the parameters, and generates updated global model parameters.

[0111] 3) Repeat the distribution of model parameters and iterative training until the preset stop condition is met.

[0112] It can be seen from this embodiment that the empathy dialogue training method and system based on scenario adaptation of the present invention enable participants A and B to jointly train a dialogue generation model to generate personalized dialogue responses in the medical and travel fields; through the federated learning framework, the data privacy and security of the participants are protected, and the training efficiency is improved; the aggregation of model parameters and the distribution and update of the global model ensure the knowledge sharing and iterative optimization of the model among the participants, and improve the performance and adaptability of the dialogue generation model; at the same time, the method has broad application prospects and is also applicable to dialogue generation tasks in other fields, such as customer service dialogues, intelligent assistants, etc.

[0113] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for training empathy dialogue based on scenario adaptation, characterized in that: The following steps are involved: Step 1: Initialize the global model, which is the dialogue model; Before federated learning begins, a global model is initialized. The transformer-based pre-trained model GPT-2 is used to train the global model using a multi-round dialogue training set. First, the input text is passed through the word embedding layer to convert each word into a corresponding word vector representation. The input sequence is X = [x1, x2, ..., x n ], where x j represents the index of the jth word, n is the number of words in the input sequence; the word embedding layer converts the index x of each word j Mapped to a d-dimensional word vector Get the word embedding representation E = [e1, e2, ..., e n ]; GPT-2 uses a multi-head self-attention mechanism to capture the contextual relationship in the input sequence. The multi-head self-attention mechanism includes three steps: ① Calculation of query, key and value. The word embedding representation E of the input sequence is transformed by three matrices Q=E·WQ, K=E·WK, V=E·WV to obtain the query vector Q, key vector K and value vector V, where WQ, WK and WV are learnable parameter matrices respectively; ② Calculation of attention score. The dot product between the query vector Q and the key vector K is calculated, and then the attention score is calculated by the scaling factor d. k Scaling to get the attention score: The softmax function is a normalized exponential function that converts the result value into a probability between [0,1]; ③ Weighted summation, weighted summation of the attention score and the value vector V, to obtain the output of self-attention: Output = Attention V; In order to retain the position information in the input sequence, gpt-2 uses position encoding to add a fixed vector representation to each position. The position encoding formula is as follows: Among them, pos represents the position index, i represents the dimension index, and d represents the dimension of the word vector; after the self-attention calculation and position encoding, the input sequence is added to the output Output of the self-attention through the residual connection; after each attention layer, GPT-2 also contains a feedforward neural network layer for nonlinear transformation and mapping of the features of each position, and the feedforward neural network layer consists of two linear transformations and an activation function; the last layer of the GPT-2 model is a linear transformation and a softmax function, which is used to map the final output of the model to the probability distribution on the vocabulary: output_probs = softmax(Output·W2+b2), where W2 and b2 are the learnable parameter matrix and bias vector respectively; Step 2: Distribute the global model and set the initial global model parameters θ g Send to each client {U1,U2,...,U m }; Step 3: Local model training: send the global model parameters obtained by the server in the previous iteration to the client. g Initialize local model parameters θ i and the local reference model parameters θ i ′, each client uses its own local data set to train the local model. The data set is the conversation data set. The data sets of all clients are marked as {D1, D2, ..., D m }; Next, update the model parameters through the optimization algorithm: θ i ←θ i -η·▽L(θ i ,D i ), where θ i is the local model parameter of the i-th client, η is the learning rate, L(θ i ,D i ) is the loss function; in local model training, the gpt-2 model is optimized based on the tuning method of reinforcement learning proximal policy optimization PPO; Step 4: Parameter upload and aggregation. Each client uploads the locally trained and updated model parameters to the server. After receiving the parameters, the server aggregates the parameters through the aggregation algorithm to obtain the global model parameters θ g ; Step 5: Model update: the server updates the aggregated global model parameters θ g After receiving the global model parameters, the client uses them as the local model parameters for the next round. After replacing the local model with the global model, the original local model parameters are used as the reference model parameters, i.e., θ i =θ g ,θ i ′=θ i-1 ; Step 6: Iterative training, repeating steps 3 to 5 until the preset stopping condition is met, that is, the maximum number of iterations is reached or the model converges; The specific process of optimizing the gpt-2 model is as follows: (1) Initialize the local model, θ i =θ g At the same time, if it is the first time to receive the global model from the server, the local reference model parameters θ′ are initialized using the global model parameters i =θ g Otherwise, use the model parameters of the local model at the previous moment to assign θ i ′=θ i-1 ; Then the local data set samples X'={x1',x2',...,x n '} is input into the local model, and action a is executed to obtain the output Y = {y1, y2, ..., y n }, where y i Contains input x i '; (2) {y1,y2,...,y n }Input into the reward model, the sentiment analysis model is used as the reward model to determine whether it is similar to the above sentiment, and the score r is obtained i ; (3) The generation process of the gpt-2 model is regarded as a policy network, that is, mapping the input sequence to the probability distribution of the output sequence; using the current policy network to generate a batch of local data set samples, and calculating the log-likelihood value of each local data set sample; for the local data set sample X', the output sequence generated by the policy network is Y, and the log-likelihood value is calculated: log_probs = logP(Y|X',θ), where θ is the parameter of the policy network; in order to use the PPO algorithm for policy optimization, the advantage estimate of each sample is calculated to measure the improvement of the current policy over the old policy; the advantage estimate is calculated using the value function-based method GAE, which is the weighted average of multiple value estimates, and the advantage estimate value at time t is estimated based on time t+1: in, is the advantage estimate at time t, is the temporal difference error of the sample at time t+1, γ is the discount factor, and λ is the GAE parameter; The objective function is optimized using the proximal strategy optimization penalty method. The objective function is as follows: Among them, s t is the state, a t For action, is the sampled state s under the π strategy at time t t and action a t The expected value, p θ (a t |s t ) is the probability of taking action in state θ at time t, p θ′ (a t |s t ) is the probability of the action in the state θ′ at time t, θ is the parameter of the policy network to be optimized, and θ′ refers to the parameter sampling data of the action generated by the actual interaction with the environment, from which the state s is sampled. t 、Action a t Then, calculate its advantage estimate A θ′ (s t ,a t ), the ultimate goal is to use θ′ to adjust θ; the proximal strategy is used to optimize the penalty, as shown in the following formula: Where β is the penalty term, KL(θ,θ′) refers to the KL divergence value of θ and θ′, and β is dynamically adjusted using the adaptive KL penalty; if KL(θ,θ′)<KLmax, increase β; if KL(θ,θ′)>KLmax, reduce β, and KLmax refers to the maximum value of the set KL divergence; (4) Strategy optimization: Use the gradient descent method to optimize the parameters θ of the policy network to maximize the optimization objective function; by calculating the gradient of the objective function with respect to θ Update the parameters of the policy network: Among them, α is the learning rate; (5) Iterative optimization: repeat steps (1) to (4) to perform multiple rounds of policy evaluation and optimization. In each round of iteration, by generating new samples and calculating advantage estimates, the parameters of the policy network are updated, and the generation performance of the model is gradually improved.

2. The method according to claim 1, characterized in that In step 1, the gpt-2 model training also uses the cross entropy function based on maximum likelihood estimation and performs parameter optimization through the gradient descent algorithm.

3. The method according to claim 1, characterized in that PPO is an optimization algorithm based on policy gradient. It tries to calculate a new strategy in each iteration to minimize the loss function while ensuring that the deviation from the strategy of the previous iteration is relatively small.

4. The method according to claim 1, characterized in that The dialogue data sets include a medical dialogue data set and a hotel reservation dialogue data set.

5. A system for implementing the method according to any one of claims 1 to 4.

Images