Chinese co-empathetic sentence training method and system based on UniLM model and copy mechanism

Abstract

The present application belongs to the technical field of natural language generation for Chinese, and proposes a Chinese empathetic sentence training method and system based on UniLM model and Copy mechanism. In view of the lack of context relevance of empathetic reply and user input, and the particularity of the reply containing emotional keywords and complex event details, the Copy mechanism is used to enable the output to copy emotional keywords and complex event details from the input. At the same time, in view of the lack of sufficient and diverse training corpus, the generated empathetic reply is comprehensively evaluated, and the high-quality empathetic reply meeting the expected standard and the user input are put into the original training corpus for compound automatic iterative training to enhance the training data. The present application fuses the Copy mechanism in the decoder to copy emotional keywords and complex event details to the output to solve the problem. In view of the lack of empathetic corpus for Chinese psychological dialogue, the present application adopts compound automatic iterative training to enhance the training data.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language generation for Chinese, and particularly relates to a Chinese empathetic reply generation method based on UniLM and a Copy mechanism. BACKGROUND

[0002] With the application of deep learning in various fields, intelligent conversation systems have also developed rapidly. Users hope to achieve emotional communication with intelligent conversation systems, and empathy can achieve this purpose. Therefore, empathetic reply generation has emerged. Empathy is defined by Carl Ransom Rogers as: in the process of interpersonal communication, standing in the position of others to imagine their experiences and logic, experiencing their thoughts and feelings, and looking at problems and solving problems from the perspective of others. Empathetic reply generation refers to that an intelligent conversation system judges the emotional state of a user through historical conversations, and generates an emotional reply that understands the user's feelings. Existing research shows that an intelligent conversation system with empathy can not only improve user satisfaction, but also obtain more positive feedback from users.

[0003] In psychological health counseling conversations, an intelligent conversation system as an auxiliary tool can help counselors solve part of the tasks, and is considered to be the key to psychological health intervention, auxiliary counseling diagnosis and other service applications. Therefore, an intelligent conversation system endowed with empathy has gradually become a research hotspot. A good conversation model must have strong context relevance between its input and output. Context relevance refers to the mutual relationship between the input of a user and the output of a model. At present, the mainstream reply generation method is a sequence-to-sequence method based on deep learning or a pre-trained model.

[0004] The traditional encoder end of sequence-to-sequence is mainly RNN, LSTM and the like. Compared with the Transformer, RNN, LSTM and the like are not good at semantic feature extraction, and have some shortcomings in long-distance dependence. Although the replies generated by various language models based on the Transformer are more readable than RNN, LSTM and the like, there is still a problem of context irrelevance caused by inaccurate generation details. SUMMARY

[0005] In view of the problems in the prior art, the application provides a Chinese empathetic sentence training method based on a UniLM model and a Copy mechanism.

[0006] This invention is implemented as follows: a Chinese empathic response generation method based on the UniLM model and the Copy mechanism. The purpose of integrating the Copy mechanism is to copy emotional keywords and complex event details from the source sequence into the output. Then, evaluation criteria such as perplexity are used to evaluate the output empathic responses. Responses that meet expectations and user statements are put into the original training corpus for repeated automatic iterative training to obtain a further updated and optimized empathic response generation model.

[0007] The technical solution adopted in this invention is a Chinese empathic response generation method based on the UniLM model and the Copy mechanism, which specifically includes the following steps:

[0008] Step 1: Use web crawling technology to crawl corpora with empathic capabilities in the field of psychological dialogue, and preprocess them to obtain input representations;

[0009] Step 2: Pre-train the UniLM model, using three types of language models, each with a different self-attention masking mechanism.

[0010] Step 3: Calculate the loss using the cross-entropy loss function, complete the pre-training of the UniLM model, and obtain the empathic response generation model;

[0011] Step 4: Based on the UniLM model, perform the empathic response generation task, and decode the vocabulary probability distribution through the self-attention mechanism of the sequence-to-sequence language model.

[0012] Step 5: Based on Step 4, construct a decoder that includes a copy mechanism, introduce generation probability and copy probability, and optimize the vocabulary probability distribution in Step 4;

[0013] Step 6: Use the cross-entropy loss function as the model's loss function and apply the BeamSearch algorithm to obtain the generated empathic responses;

[0014] Step 7: The generated high-quality empathic responses and user statements are put into the corpus from Step 1, and further refined through repeated automatic iterative training based on the UniLM model to obtain the updated and optimized empathic response generation model.

[0015] Furthermore, each input consists of two text sequences, Segment1 (denoted as s1) and Segment2 (denoted as s2), for example: "[CLS] I always think about people or things I really hate [SEP] I understand that you are confused and perplexed because you are preoccupied with negative events in your life and forget positive events [SEP]". [CLS] marks the beginning of the sequence, and [SEP] marks the end of the sequence. The text sequence pairs are represented by three types of embeddings.

[0016] Further, the UniLM model is stacked by 12 layers of Transformer structure, each layer of the Transformer has 768 hidden nodes in the hidden layer and 12 heads, and has the same structure as BERT-BASE, and thus can initialize parameters from the trained BERT-BASE model. The UniLM model can simultaneously complete three pre-training objectives, and can complete the prediction tasks of the unidirectional training language model, the bidirectional training language model, and the sequence-to-sequence language model, so that the model can be applied to the natural language generation task. Different MASK mechanisms are adopted for different language models. The MASKING mode is as follows: the overall proportion is 15%, in which 80% of the cases are directly replaced by [MASK], 10% of the cases are randomly selected to replace a word in the dictionary, and the last 10% of the cases are not processed. In addition, 80% of the cases are to mask one word each time, and 20% of the cases are to mask two words bigram or three words trigram. For the MASK to be predicted, the unidirectional language model uses the context on one side, for example, to predict the mask in the sequence "X1X2[MASK]X4", only the information of X1, X2 and itself is available, and the information of X4 is unavailable. The bidirectional language model encodes the context information from two directions, for example, in "X1X2[MASK]X4", the information of X1, X2, X4 and itself is available. In the sequence-to-sequence language model, if the MASK is in S1, only the context information of S1 can be encoded; if the MASK is in S2, it can obtain the context information to the left of the MASK, including S1.

[0017] Further, the text representation output by the Transformer network is input into a Softmax classifier to predict the masked word, and a cross-entropy loss function is used for the predicted word and the original word to optimize the model parameters and complete the pre-training.

[0018] Further, by randomly masking a certain proportion of the words in the target sequence, the sequence-to-sequence language model is used to learn to restore the masked words, and the training objective is to maximize the probability of the masked words based on the context information. The [SEP] at the end of the target sequence can also be masked, so that the model learns when to terminate the generation of the target sequence. The model uses the MASK mechanism combined with the attention mechanism to obtain the text feature vector, which is input into the full connection layer to obtain the word table probability distribution.

[0019] Further, the word table probability distribution is input into the full connection layer and the Sigmoid layer to obtain the generation probability. The copy probability is introduced, and the updated and improved word table probability distribution is obtained by combining the generation probability and the copy probability.

[0020] Further, the cross-entropy loss function is used to complete the fine-tuning task of the model, and the Beam Search algorithm is used to generate the empathetic reply.

[0021] Further, four evaluation indexes such as perplexity, BLEU-4, F1 and expert evaluation are used to comprehensively evaluate the empathetic reply generated in step 6, and the empathetic reply meeting the expected standard and the user input are automatically put into the original corpus in step 1 for multiple automatic iterative training, so as to enhance the training data and obtain an updated and optimized Chinese empathetic reply generation model.

[0022] The purpose of the present application is to solve the problem that the empathetic reply generated based on the Transformer network cannot generate emotional keywords and complex event details, and to solve the problem by fusing the Copy mechanism in the decoder and copying the emotional keywords and complex event details to the output.

[0023] Another purpose of the present application is to solve the problem that the corpus of Chinese empathetic dialogue has a lack of empathetic ability, and the present application uses multiple automatic iterative training to enhance the training data.

[0024] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present application are analyzed from the following aspects:

[0025] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty in solving the problems, the technical solutions to be protected by the present application and the results and data in the research and development process are closely combined to analyze in detail and profoundly how the technical solutions solve the technical problems and bring some creative technical effects after solving the problems. The specific description is as follows:

[0026] In the process of interpersonal communication, people more hope to stand on the position of others to imagine the experience and logic of others, understand the thoughts and feelings of others, and solve problems from the perspective of others. Among them, the intelligent conversation system endowed with the ability of empathy has gradually become a research hotspot. The present application solves the generation of empathetic reply, which means that the intelligent conversation system judges the emotional state of the user through the historical conversation, and generates an emotional reply that understands the user's feelings. The intelligent conversation system with the ability of empathy not only can improve the satisfaction of the user, but also can obtain more positive feedback from the user.

[0027] Secondly, from the perspective of the product as a whole, the technical effects and advantages of the technical solutions to be protected by the present application are described as follows:

[0028] The application proposes a Chinese empathy reply generation method based on a UniLM model and a Copy mechanism. The UniLM model is used as the basic architecture, and the Copy mechanism is fused in the decoder to solve the problem that the empathy reply generated based on the Transformer network cannot generate emotional keywords and complex event details. For the problem of lack of empathy-competent corpus of Chinese mental dialogue, the application uses multiple automatic iterative training to enhance the training data.

[0029] The application copies emotional keywords and complex event details in the source sequence to the output, and then evaluates the empathy reply output using evaluation criteria such as perplexity, and puts the expected reply and user statement into the original training corpus for multiple automatic iterative training to obtain a further updated and optimized empathy reply generation model. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a framework diagram of the Chinese empathy reply generation model based on the UniLM model and the Copy mechanism provided by the embodiment of the application.

[0031] Figure 2 is a UniLM model architecture schematic diagram used by the embodiment of the application.

[0032] Figure 3 is a specific flowchart of the Chinese empathy reply generation method based on the UniLM model and the Copy mechanism provided by the embodiment of the application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the application clearer and more apparent, the application will be further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0034] In order for those skilled in the art to fully understand how the application is specifically implemented, this part is an explanatory embodiment of the description and explanation of the technical scheme of the claims.

[0035] The Chinese empathy reply generation method based on the UniLM model and the Copy mechanism is further described in detail in combination with the drawings and the specific embodiments.

[0036] As shown in Figure 1 The application mainly uses the UniLM model as the basis, and fuses the Copy mechanism in the decoding end to achieve the purpose of fully utilizing the context correlation of complex event details in the conversation empathy. It mainly includes four stages of input processing, pre-training, empathy reply generation and multiple training. The specific embodiments are as follows:

[0037] The pre-trained corpus includes the statements of psychological problems of the psychological counseling visitors and the empathetic replies of the counselors. The statement of the visitor is Segment 1, denoted as S1, and the reply of the counselor is Segment 2, denoted as S2. The special marks [CLS] and [SEP] are added, and the input representation of the model is as shown in the following formula: “[CLS] S1 [SEP] S2 [SEP]”. Figure 2 As shown in the formula, the input representation of the model is composed of the sum of Segment Embedding, Position Embedding and Token Embedding.

[0038] The model is pre-trained, the input Embedding vector is encoded by each layer of the Transformer, the input vector is aggregated by using the multi-head attention mechanism, the range of attention of each word or position is controlled by the mask matrix, the attention distribution of the current position to other positions is obtained, and the feature vector of the current position of the decoder is calculated.

[0039] The attention distribution At of the generated word vector to the text feature vector XInput at time t is as follows:

[0040]

[0041] The feature vector XOutput output by the decoder at time t is as follows:

[0042] X Output =A t *W v *X Intput

[0043] Wherein, Xt is the target vector at time t; XInput is the text feature vector at time t; M is the mask matrix, which controls the range of word attention; dk is the dimension of the word vector; Wq, Wk and Wv are learning parameters.

[0044] The Softmax function maps the vector of scores s to a probability distribution, and its definition is as follows:

[0045]

[0046] Wherein, i represents the number of output nodes; si is the output value of the ith node; n is the number of output nodes, i.e. the number of categories of classification.

[0047] Further, the cross-entropy loss is calculated between the model prediction result XOutput, denoted as s, and the original segmented word st to optimize the parameters of the model. The cross-entropy function is defined as follows:

[0048]

[0049] Preprocessing process: the preprocessed data is input into the model for training, a total of 20 Epochs, Dropout is 0.1, hidden vector dimension is 768, learning rate Learning_rate is 2e-5, Epochs is 20, batch size Batch_size is 32, attention head number is 12, hidden layer number is 12, embedding layer is 12, hidden layer unit number is 768, vocabulary size is 21128. The maximum input length is set to 512, the maximum generated empathetic reply length is set to 40, and the loss is calculated using the cross-entropy function.

[0050] After pre-training, the UniLM sequence-to-sequence language model is used for fine-tuning to perform the empathetic reply generation task. During decoding, for example: the user inputs a statement of the inner psychological problem "X1", when t = 1, the input sequence is "[CLS]X1[SEP]Y1[MASK]", and "[MASK]" is added at the end of the sequence, which corresponds to the feature representation of the next word. "[CLS]X1[SEP]" is the known source sequence, which can see the context information within the sentence during the encoding stage. "Y1[MASK]" is the predicted target sequence, which can see the information of the source sequence and the left part of the target sequence during the decoding stage. The model fuses the encoder and decoder together through the mask matrix.

[0051] After the corpus sample is encoded by the UniLM model, a sequence length X hidden size matrix is obtained, the first row is the feature representation of [CLS], the second row is the feature representation of X1, and so on. During the decoding stage, the [MASK] feature representation is passed through a linear layer, and a Softmax function is used to obtain the probability distribution of the words in the vocabulary, and the word with the highest probability is selected as the decoded word. Repeat the above steps, stop when [SEP] is generated, and obtain the feature vector XOutput output by the decoder at time t. The specific calculation is as follows:

[0052] XOutput is subjected to two linear transformations and a Softmax function to obtain the word table probability distribution Pv.

[0053] P v =Softmax(W ′ (W*X Output +b)+b ′ )

[0054] Where W ′ , W, b, b ′ are learnable parameters.

[0055] We introduce a generation probability Pg, representing the probability of generating a word from the vocabulary; and a copy probability Pc, representing the probability of copying a word from the source text, where Pg + Pc = 1. Pg is calculated by passing XOutput, At, and Xt through a fully connected layer and a sigmoid function.

[0056] P g =Sigmoid(W[X t X Output A t ]+b)

[0057] Where W and b are learnable parameters.

[0058] Further calculations were performed on the updated and improved vocabulary probability distribution:

[0059] P(w)=P g *P v (w)+P c *A t

[0060] Where w is not a word in the vocabulary, P v When (w) = 0, the predicted word is generated from the source sequence; when w is not a word in the source sequence, A... t =0, the predicted words are generated from the vocabulary. The Copy mechanism copies sentiment keywords and complex event details (high-probability words) from the source sequence as part of the generated empathic response, which can control the accuracy of the generated empathic response to some extent. The Copy mechanism also plays a role in dynamically expanding the vocabulary to reduce the probability of generating out-of-vocabulary words.

[0061] With the Beam size set to 1, the Beam Search algorithm is used to search for near-optimal target sequences to generate empathic responses. The generated empathic responses are evaluated, and the compliant empathic responses, along with the user's statements, are fed into the original corpus for repeated automatic iterative training to enhance the training data, resulting in an updated and optimized Chinese empathic response generation model.

[0062] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for generating Chinese empathic responses based on the UniLM model and the Copy mechanism, characterized in that, The emotional keywords and complex event details in the source sequence are copied into the output; the output empathic responses are evaluated using the perplexity evaluation criterion; responses that meet the expectations and user statements are put into the original training corpus for repeated automatic iterative training to obtain a further updated and optimized empathic response generation model. Specifically, the steps include the following: Step 1: Use web crawling technology to crawl corpora with empathic capabilities in the field of psychological dialogue, and preprocess them to obtain input representations; Step 2: Pre-train the UniLM model, using three types of language models, each with a different self-attention masking mechanism. Step 3: Calculate the loss using the cross-entropy loss function, complete the pre-training of the UniLM model, and obtain the empathic response generation model; Step 4: Based on the UniLM model, perform the empathic response generation task, and decode the vocabulary probability distribution through the self-attention mechanism of the sequence-to-sequence language model. Step 5: Based on Step 4, construct a decoder that includes a copy mechanism, introduce generation probability and copy probability, and optimize the vocabulary probability distribution in Step 4; Step 6: Use the cross-entropy loss function as the model's loss function and apply the BeamSearch algorithm to obtain the generated empathic responses; Step 7: The generated high-quality empathic responses and user statements are put into the corpus of Step 1, and further refined based on the UniLM model for repeated automatic iterative training to obtain the updated and optimized empathic response generation model; Step 2 specifically includes: initializing parameters with the BERT-BASE pre-trained model; based on the same Transformer network structure, using the prediction of different masks as the pre-training target, completing the prediction tasks of unidirectional, bidirectional, and sequence-to-sequence language models, and evenly distributing the use of different language models.

2. The Chinese empathic response generation method based on the UniLM model and Copy mechanism as described in claim 1, characterized in that, Step 4 specifically includes: using the self-attention masking mechanism of the sequence-to-sequence language model, randomly masking the word segments in the target sequence and masking the end of the sequence to learn when to stop generating empathetic responses; taking the maximization of word segmentation probability given contextual information as the training objective, using the masking mechanism to fuse encoding and decoding, and combining it with the attention mechanism to obtain text feature vectors; inputting the decoded feature vectors into a fully connected layer, and using the Softmax function to obtain the vocabulary probability distribution.

3. The Chinese empathic response generation method based on the UniLM model and Copy mechanism as described in claim 1, characterized in that, Step 5 specifically includes: inputting the vocabulary probability obtained in the previous step into the fully connected layer and the Sigmoid layer to obtain the generation probability, then introducing the copy probability, and fusing the generation probability and the copy probability to obtain the updated and improved vocabulary probability distribution.

4. The Chinese empathic response generation method based on the UniLM model and Copy mechanism as described in claim 1, characterized in that, Step 7 specifically includes: evaluating the empathic responses generated in step 6 using a perplexity evaluation criterion; automatically incorporating the expected empathic responses and user input into the corpus from step 1 for iterative training to enhance the training data and obtain an updated and optimized empathic response generation model.

5. A Chinese empathic response generation system based on the generation method of any one of claims 1-4, characterized in that, include: The detail copy module is used to copy emotional keywords and complex event details from the source sequence into the output; The empathic response generation model module is used to evaluate the output empathic responses using a perplexity evaluation criterion. Responses that meet expectations and user statements are fed into the original training corpus for repeated automatic iterative training to obtain a further updated and optimized empathic response generation model.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the Chinese empathic response generation method based on the UniLM model and Copy mechanism as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the Chinese empathic response generation method based on the UniLM model and Copy mechanism as described in any one of claims 1-4.

8. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the Chinese empathy response generation system as described in claim 5.

Images