Chinese multi-turn dialogue model

Abstract

The application provides a Chinese multi-round dialogue model, taking GPT-2 as a main framework, fusing a dialogue history keyword extraction model and a dialogue theme switching discrimination model, so that the Chinese multi-round dialogue model can discriminate whether to switch the chat theme when generating a reply sentence, and consider whether to copy a word from the dialogue history or select a word from a global word table according to a generation mechanism, thereby improving the multi-round dialogue effect, enhancing the relevance of the context, and overcoming the difficulty of answering irrelevant questions in the open domain dialogue.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a Chinese multi-turn dialogue model. Background Technology

[0002] As my country's population ages, many elderly people are experiencing loneliness and other problems, making emotional companionship for the elderly an increasingly important issue. In recent years, emotional companionship robots have emerged, with chatbots being one type. These robots can replace children in accompanying the elderly, alleviating their loneliness, enriching their lives, and providing them with psychological comfort.

[0003] Based on their performance, chatbot dialogue systems can be categorized into single-turn and multi-turn systems. Single-turn systems only consider the current user input, without considering context, referential meaning, omissions, or hidden information. Multi-turn systems, on the other hand, need to comprehensively consider the dialogue context (including historical dialogue information and queries), establish long-term dependencies, and provide responses that better align with the dialogue logic. Multi-turn dialogue also involves a "decision-making" process, making it more intelligent and applicable to a wider range of scenarios compared to single-turn dialogue. Multi-turn dialogue is more suitable for real-world applications. Open-domain multi-turn dialogue generation tasks often produce more diverse and fluent responses; however, because they easily fit general language models during pre-training, the model struggles to control the direction of its generation. Multi-turn dialogue can leverage historical topic information to assist in generating the current response. Historical topic information has the following characteristics: a hierarchical structure from words to sentences and sentences to fragments; much redundant content irrelevant to the current response; long-distance dependencies between words or sentences; and changes in content or shifts in semantic relationships. However, effective extraction of historical topic information and utilization of its relevant content are crucial for generating effective responses. Words with distinctive features in the dialogue history have a significant impact on the model's performance during training. Therefore, a dialogue history key extraction network is constructed to obtain keywords from the dialogue history's topic information for dialogue generation. Furthermore, multi-turn dialogue models sometimes fail to provide correct responses to topic changes, leading to the failure of Chinese multi-turn dialogues, which is a major challenge in open-domain generative dialogue. Simply concatenating the dialogue history's topic information as input to the model, to some extent, ignores the dynamic changes in chat topics and is not conducive to the dialogue model's ability to grasp topic transitions. Summary of the Invention

[0004] The purpose of this invention is to propose a Chinese multi-turn dialogue model to improve the effect of multi-turn dialogue, enhance the relevance of context, and overcome the problem of irrelevant answers in open-domain dialogue.

[0005] To achieve the above objectives, this invention proposes a Chinese multi-turn dialogue model, which uses GPT-2 as the main framework and integrates a dialogue history keyword extraction model and a dialogue topic switching discrimination model.

[0006] The dialogue history keyword extraction model includes a generation mode and a copy mode. The generation mode infers the meaning of the original sentence and selects a word from the global list based on the current state. The copy mode calculates the attention distribution and converts it into the probability of the vocabulary. Based on the probability information of the original sentence, it copies the required words from the original sentence to the target output.

[0007] The dialogue history keyword extraction model divides the vocabulary into an input vocabulary set and an output vocabulary set. During decoding, if a word is unique to the input vocabulary set, its generation probability is 0, and its copy probability remains unchanged. If a word is unique to the output vocabulary set, its copy probability is 0, and its generation probability remains unchanged, meaning the model performs a normal generation task. If a word exists in both the input and output vocabulary sets, its probability is determined by the sum of its generation and copy probabilities, resulting in the following final generation probability:

[0008]

[0009] In the formula, For words The generation probability, For words The probability of replication;

[0010] The dialogue topic switching discrimination model determines whether a topic should be switched by judging the degree of correlation between the semantics of each utterance and the semantic information in the entire dialogue history.

[0011] Furthermore, the Chinese multi-turn dialogue model adopts a planar mode dialogue history topic information modeling method, which concatenates the dialogue history into a long text sequence and uses segmentation vectors to distinguish them.

[0012] Furthermore, given a standard response Dialogue on historical themes Each statement in the dialogue history topic information Depend on Composed of individual characters, a generative dialogue model is trained using maximum likelihood estimation: ;

[0013] Given the dialogue history topic information U and the generated ,…, Generate the first dialogue using a pre-trained generative dialogue model. The first sentence The generation of the reply statement ends when a terminator is generated in the entire sentence.

[0014] Furthermore, in historical keyword extraction models, sequence-to-sequence text generation models incorporate an attention mechanism during decoding when handling generation or translation tasks. This captures long-distance dependencies and positional information of the original text at the encoding stage, and during encoding, word embeddings are encoded to obtain the state. At the decoding end, the decoder receives the reference summary terms sequentially during the training phase, and at time step... Get decoding status ;use and The original text obtained the time step. Attention coefficients for each word, resulting in attention information. and Weighted summation yields the important context vector. :

[0015] ;

[0016] ;

[0017] ;

[0018] Keyword extraction network model obtained , and decoder input To calculate the generation probability :

[0019] ;

[0020] in , , These are model parameters. It is a bias parameter. It is the Sigmoid activation function;

[0021] After calculating the generation probability of the model, the probability distribution of the duplicate word is calculated using the attention distribution. Finally, the output probability of the word in the decoder is:

[0022] ;

[0023] In the formula, Words that represent the original document, when When it does not appear in the output vocabulary set, =0; when Not found in the input vocabulary set. It is 0.

[0024] Furthermore, to overcome the problem of duplicate text generation caused by keyword duplication, the network superimposes the attention weights from previous time steps to construct a new structure. Use previous attention weights to influence current attention weights:

[0025] ;

[0026] ;

[0027] ;

[0028] in This is the loss during model training; if a word has already been generated, then... It will be relatively large. Part of the loss is Therefore, it will reduce the time required for model training. The value, i.e., reducing the value of By focusing on location, the problem of repeated generation can be solved to some extent.

[0029] Furthermore, Top-p and Top-k sampling methods are used to optimize the logic of the dialogue model decoder.

[0030] Furthermore, the dialogue topic switching discrimination model considers all previous historical context information. To represent the themes of historical dialogue In the response generation phase of the dialogue model, the current input statement is determined by a discriminative model. semantic information Information related to historical themes The degree of conformity; if the current input conforms to the historical topic information, the historical topic information and the current input statement together guide the generation of a response; if the current input statement does not conform to the historical topic information, it indicates that a new topic dialogue will be initiated and the historical dialogue will be cleared;

[0031] The GPT-2 encoded number is obtained through the GPT-2 model. Historical context information for each discourse point:

[0032] ;

[0033] ;

[0034] Use all dialogue contexts As input, therefore The context of the statement for:

[0035] ;

[0036] Utilizing fully connected networks and Transform the text to obtain the topic representation information corresponding to the dialogue text;

[0037] ;

[0038] in For information transformation networks, To distinguish networks by switching themes, Switch probability information for themes.

[0039] Compared with the prior art, the advantages of the present invention are:

[0040] 1. Multi-turn dialogue systems need to comprehensively consider the dialogue context (including historical dialogue information and queries), establish long-term dependencies in the dialogue, and provide responses that are more consistent with the dialogue logic. In multi-turn dialogue, the robot dialogue system also involves a "decision-making" process, making it more intelligent than single-turn dialogue and allowing for a wider variety of application scenarios. Compared to single-turn dialogue, multi-turn dialogue is more in line with real-world application scenarios.

[0041] 2. The multi-turn dialogue model of this invention integrates the functions of dialogue history keyword extraction and dialogue topic switching discrimination, enabling the model to determine whether to switch the chat topic when generating response statements, and to consider whether to copy a word from the dialogue history or select a word from the global vocabulary according to the generation mechanism. This improves the effect of multi-turn dialogue, enhances the relevance of the context, and overcomes the problem of irrelevant answers in open domain dialogue.

[0042] 3. This invention uses Top-p and Top-k sampling methods to optimize the logic of the dialogue model decoder and improve the effect of the dialogue model in generating response statements. Attached Figure Description

[0043] Figure 1 This is a structural diagram of a Chinese multi-turn dialogue model in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the generation of Chinese multi-turn dialogue responses in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.

[0046] This invention proposes a Chinese multi-turn dialogue model, such as... Figure 1As shown, using GPT-2 as the main framework, the model integrates dialogue history keyword extraction and dialogue topic switching discrimination functions, enabling it to determine whether to switch chat topics when generating response statements, and to consider whether to copy a word from the dialogue history or select a word from the global vocabulary according to the generation mechanism. This improves the effect of multi-turn dialogue, enhances the relevance of the context, and overcomes the problem of irrelevant answers in open domain dialogue.

[0047] The model's input consists of dialogue history and the current sequence of input statements. Represents the input word vectors, This represents the output of the last layer of the model. It is the probability of the original generated words in the dialogue model, therefore the extraction probability of the dialogue history keyword extraction network. for:

[0048] ;

[0049] in, The value range of is [0,1]. and These are learnable parameters. It is the Sigmoid activation function. and The input data dimension is the word vector dimension.

[0050] The probability of obtaining the original generated words in the dialogue model Extraction probability of keyword extraction network Finally, the probability distribution output is:

[0051] ;

[0052] in, It is the probability of the original words generated by the dialogue model. This represents the attention score of the last layer of the model, and its data dimension is the size of the vocabulary. Vocabulary used in historical dialogues.

[0053] In generative multi-turn dialogue models, when the topic changes during dialogue generation, the response content is still generated based on the information from the previous topic, resulting in irrelevant answers. Therefore, it is necessary to construct a dialogue topic switching discrimination model to construct more effective historical topic information to guide the model in generating appropriate response statements.

[0054] Specifically, the following designs are included:

[0055] 1. The Construction of Chinese Multi-Turn Dialogue

[0056] The invention employs a planar dialogue history topic information modeling method, which concatenates the dialogue history into a long text sequence and uses segmentation vectors for differentiation. Specifically, the [CLS] character is used to mark the start of the text, and the [SEP] character is used after each paragraph to indicate the end of the paragraph. Speaker embeddings represent different speakers. For a dialogue, the concatenation process can be as follows: [CLS]Have you eaten? [SEP]Yes! [SEP]What did you eat? [SEP]Fish, shrimp, crab, shellfish [SEP], and so on. Figure 2 As shown. Then, the above concatenation result is used as input to the generative dialogue model, allowing the model to undergo autoregressive training.

[0057] At each time step, the self-attention mechanism of the GPT-2-based Chinese multi-turn dialogue model can only observe information from the left. Given a standard response... Dialogue on historical themes Each statement in the dialogue history topic information Depend on Composed of several characters, our goal is to train a generative dialogue model using maximum likelihood estimation:

[0058] ;

[0059] Given the dialogue history topic information U and the generated ,…, Generate the first dialogue using a pre-trained generative dialogue model. The first sentence The generation of the reply statement ends when a terminator is generated in the entire sentence.

[0060] 2. Keyword extraction model for dialogue history

[0061] The model operates in two modes during decoding: generation and copying. Generation mode primarily infers meaning from the original sentence, selecting a word from the global vocabulary based on the current state. Copying mode calculates the attention distribution and converts it into a probability-like word list, then copies the required words from the original sentence to the target output based on the probability information. Here, we divide the vocabulary into two parts: the input vocabulary set and the output vocabulary set. In most cases, these two sets intersect; the part outside the union of the two sets represents unknown words. During decoding, if a word is unique to the input vocabulary set, its generation probability is 0, and its copying probability remains unchanged. If a word is unique to the output vocabulary set, its copying probability is 0, and its generation probability remains unchanged, meaning the model performs a normal generation task. If a word exists in both the input and output vocabulary sets, its probability is determined by the sum of its generation and copying probabilities. Therefore, combining the traditional generation and copying modes to build a new model can improve text generation performance and increase generation quality. The final generation probability of the model is:

[0062] ;

[0063] in, For words The generation probability, For words The probability of replication.

[0064] Sequence-to-sequence text generation models often incorporate attention mechanisms during decoding to capture long-distance dependencies and positional information in the original text at the encoding stage. During encoding, word embeddings are encoded to obtain the state. At the decoding end, the decoder receives the reference summary terms sequentially during the training phase, and at time step... Get decoding status .use and The original text obtained the time step. Attention coefficients for each word, resulting in attention information. and Weighted summation yields the important context vector. :

[0065] ;

[0066] ;

[0067] ;

[0068] Keyword extraction network model obtained , and decoder input To calculate the generation probability :

[0069] In the formula: At time step t, the original text's first... The attention coefficient of the i-th word is a scalar value used to measure the attention of the i-th word. The importance of each word is calculated through nonlinear transformation;

[0070] : The transpose of the learnable parameter vector, used for linear transformation of attention coefficients;

[0071] , Learnable weight matrices are used to transform the encoding state. and decoding state ;

[0072] Encoding end The hidden state of the nth word represents the nth word in the input sequence. The encoded output of each position;

[0073] The decoding state at time step t represents the context information at the current moment during the generation process.

[0074] Bias parameters of the attention mechanism;

[0075] Attention distribution vector, processed by the softmax function The weight of each input word is obtained by normalization.

[0076] : Represents the attention coefficient vector at time step t; it is a real number vector containing each position in the input sequence. Attention coefficient ;Specifically, Where n is the length of the input sequence, each The result is calculated from the first line of the formula:

[0077] : No. The attention weight of each word at time step t is An element of the vector represents the current generated pair of input words. The level of attention;

[0078] Context vector, which is the encoded state. The weighted sum is used to capture important information in the input;

[0079] ;

[0080] in , , These are model parameters. It is a bias parameter. It is the Sigmoid activation function; , , The transpose of the learnable weight vector corresponds to the context vector. Decoding status and decoder input The linear transformation parameters;

[0081] After calculating the generation probability of the model, the probability distribution of the duplicate word is calculated using the attention distribution. Finally, the output probability of the word in the decoder is:

[0082] ;

[0083] in Words that represent the original document, when When it does not appear in the output vocabulary set, =0; when Not found in the input vocabulary set. It is 0.

[0084] In addition, to overcome the problem of duplicate text generation caused by keyword duplication, the network superimposes the attention weights from previous time steps to construct a new structure. Use previous attention weights to influence current attention weights:

[0085] ;

[0086] ;

[0087] ;

[0088] In the formula, Learnable parameter vectors used to modulate the effect of cumulative attention;

[0089] : No. The attention weight of each word at time step t is An element of a vector;

[0090] : No. The cumulative attention weight of each word is An element of a vector;

[0091] in This is the loss during model training. If a word has already been generated, then... It will be relatively large. Part of the loss is Therefore, it will reduce the time required for model training. The value, i.e., reducing the value of By focusing on location, the problem of repeated generation can be solved to some extent.

[0092] 3. Dialogue Topic Switching Discrimination Model

[0093] The dialogue topic switching discrimination model determines whether a topic switch has occurred by judging the correlation between the semantics of each utterance and the semantic information throughout the entire dialogue history. Let... This indicates the entire dialogue. Indicates the first The historical dialogue of a sentence is represented as follows: Current input statement ,in Indicates the first The first sentence One word, representing By judging and The differences between them define how the topic of the conversation changes.

[0094] The dialogue topic switching discrimination model first encodes the dialogue history, based on all previous historical context information. To represent the themes of historical dialogue In the response generation phase of the dialogue model, the current input statement is determined by a discriminative model. semantic information Information related to historical themes The degree of relevance. If the current input matches the historical topic information, the historical topic information and the current input statement together guide the generation of a response; if the current input statement does not match the historical topic information, it indicates that a new topic dialogue will be initiated, and the historical dialogue will be cleared.

[0095] The GPT-2 encoded number is obtained through the GPT-2 model. Historical context information for each discourse point:

[0096] ;

[0097] ;

[0098] Use all dialogue contexts As input, therefore The context of the statement for:

[0099] ;

[0100] Utilizing fully connected networks and The transformation is performed to obtain the topic representation information corresponding to the dialogue text.

[0101] ;

[0102] in For information transformation networks, To distinguish networks by switching themes, Switch probability information for themes.

[0103] 4. Decoding mechanism settings

[0104] In previous dialogue generation tasks, greedy search strategies based on maximizing output probability and bundle search strategies were commonly used for decoding. These methods often resulted in bland, repetitive, and disjointed text output by the model. Furthermore, human speech is often unpredictable; the words spoken are not always the most probable words in the language model. Bundle search, on the other hand, tends to select the words that best fit the language model, resulting in commonplace vocabulary and unoriginal text. Top-p sampling effectively overcomes this problem. At different time steps, the size of the candidate word set dynamically changes as the probability distribution of the decoded words varies. Since the decoded words in Top-p sampling are selected from the top candidate set, this dynamic adjustment allows the generated sentences to maintain both diversity and fluency. Therefore, this invention uses Top-p and Top-k sampling methods to optimize the logic of the dialogue model decoder and improve the performance of the generated response sentences. The model is trained on a dataset of 500,000 multi-turn Chinese casual conversations. The hardware configuration of the training system is shown in Table 1 below.

[0105] Table 1

[0106] operating system Ubuntu 16.04 LTS GPU NVIDIA GTX3090 Video memory capacity Integrated Development Environment PyCharm CPU Xeon 2678V3 Python Version: 3.6.13 pytorch Version: 1.7.0

[0107] Table 2 shows examples of multi-turn conversations in casual chat:

[0108] Table 2

[0109] Role Response User hello Chatbot Hello hello User Did you miss me? Chatbot Miss you User Let's go eat together. Chatbot Yes, yes. User What kind of food do you want to eat? Chatbot We can all User Want some braised pork? Chatbot No, no, no, I don't eat braised pork.

[0110] Table 3 shows examples of multi-turn dialogues for knowledge retrieval:

[0111] Table 3

[0112] User Do you know Shandong University? Chatbot have no idea User Where is Shandong University located? Chatbot Shandong User Is it in Shandong? Chatbot Yes, Shandong

[0113] Examples of multi-round dialogues for sharing experiences are shown in Table 4:

[0114] Table 4

[0115] User Yesterday, my friend and I had hot pot. Have you tried it? Chatbot No, I've eaten meat before, it's especially delicious. User What did my friend eat yesterday? Chatbot Hot pot

[0116] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A Chinese multi-turn dialogue model, characterized in that, Taking GPT-2 as a main framework, a dialogue history keyword extraction model and a dialogue topic switching discrimination model are fused; The dialogue history keyword extraction model comprises a generation mode and a replication mode; the generation mode infers according to the meaning of an original sentence and selects a word from the whole global according to the current state to generate; the replication mode calculates an attention distribution and converts it into a probability of a vocabulary table, replicates according to the probability information of the original sentence, and realizes replication of a required word from the original sentence to a target output; The dialogue history keyword extraction model divides the vocabulary table into an input vocabulary set and an output vocabulary set; when a certain word is a word unique to the input vocabulary set during decoding, the generation probability of the word is 0, and the replication probability is unchanged; If a certain word is a word unique to the output vocabulary set, the replication probability of the word is 0, and the generation probability is unchanged, that is, the model performs a normal generation task; if a certain word exists in both the input vocabulary set and the output vocabulary set, the probability of the current word is determined by the sum of the generation probability and the replication probability, and the final generation probability is: ； wherein is the probability of generation of the word is the probability of copying of the word is the probability of generation of the word is the probability of copying of the word The dialogue topic switching discrimination model discriminates whether to switch the topic by discriminating the relevance of the semantic information of each utterance and the whole dialogue history.

2. The Chinese multi-round dialogue model of claim 1, wherein, The Chinese multi-round dialogue model adopts a planar mode dialogue history topic information modeling method, concatenates the dialogue history into a long text sequence, and uses a segment division vector to distinguish.

3. The Chinese multi-round dialogue model of claim 1, wherein, Given a standard reply , dialogue history topic information , wherein each sentence in the dialogue history topic information is composed of words, a generative dialogue model is trained by maximum likelihood estimation: ; In a given dialogue history topic information U and generated ,…, , the first word of the first word of the reply sentence is generated by the trained generative dialogue model until a terminal symbol is produced in the whole sentence, the generation of the reply sentence is ended.

4. The Chinese multi-round dialogue model of claim 1, wherein, In the history keyword extraction model, the sequence-to-sequence text generation model adds an attention mechanism during decoding when processing a generation or translation task, captures long-distance dependency and position information of the original text at the encoding end, and obtains a state through encoding of the word embedding at the encoding end At the decoding end, the decoder inputs the summary words one by one during the training stage, and obtains a decoding state at each time step ​​​​​​ ； ； ； The keyword extraction network model is trained to compute a generation probability , and a decoder input to compute a generation probability : In the formula: : the attention coefficient of the original text word at time step t, is a scalar value, which measures the importance of the word, and is calculated by a nonlinear transformation; : the attention coefficient of the original text word at time step t, is a scalar value, which measures the importance of the word, and is calculated by a nonlinear transformation;​ : transpose of the learnable parameter vector for the linear transformation of the attention coefficients; , : learnable weight matrices for transforming the encoding states and the decoding states ; : the hidden state of the th word, representing the encoding output of the th position in the input sequence; : decoding state of the decoding end at time step t, indicating the context information of the current time in the generation process; : bias parameters of the attention mechanism; : attention distribution vector, obtained by normalizing softmax function, representing the weight of each input word; : denotes the attention coefficient vector at time step t; It is a real-number vector containing each position in the input sequence. Attention coefficient ;Specifically, Where n is the length of the input sequence, each The result is calculated from the first line of the formula: : No. The attention weight of each word at time step t is An element of the vector represents the current generated pair of input words. The level of attention; : context vector, is a weighted sum of the encoding states to capture important information in the input; ； wherein , , are model parameters, is a bias parameter, is a Sigmoid activation function; 、 、 : transpose of a learnable weight vector, respectively corresponding to linear transformation parameters of context vectors , decoding states and decoder inputs . After calculating the generation probability of the model, the probability distribution of the replication word is calculated by the attention distribution, and the output probability of the word in the final decoder is: ； wherein, represents a word of the original document, when not present in the output vocabulary set, is 0; when not present in the input vocabulary set: is 0.

5. The Chinese multi-round dialogue model of claim 4, wherein, To overcome the problem of duplicate text generation caused by keyword duplication, the network superimposes the attention weights of previous time steps to construct The previous attention weights are used to influence the current attention weights: ； ； ； In the formula, : a learnable parameter vector used to adjust the influence of cumulative attention; : The attention weight of the t-th word at time step t is an element of the vector ​​ : the cumulative attention weight for the : first word is an element of the : vector; wherein is the loss at model training; if a word has been generated, will be large, the loss of the part is so the value of will be reduced at model training, i.e. the attention to positions is reduced, thus solving the problem of repeated generation to some extent.

6. The Chinese multi-round dialogue model of claim 4, wherein, The Top-p and Top-k sampling methods are adopted to optimize the logic of the dialogue model decoder.

7. The Chinese multi-round dialogue model of claim 1, wherein, The dialogue topic switching discrimination model is based on all previous historical context information. To represent the themes of historical dialogue During the response generation phase of the dialogue model, the current input statement is determined by a dialogue topic switching discriminant model. semantic information Information related to historical themes The degree of conformity; if the current input conforms to the historical topic information, the historical topic information and the current input statement together guide the generation of a response; if the current input statement does not conform to the historical topic information, it indicates that a new topic dialogue will be initiated and the historical dialogue will be cleared; The historical context information at the first utterance encoded by the GPT-2 model is obtained by the GPT-2 model. ； ； with all dialog context as input, therefore context of the utterance is: ； using a fully connected network on and transforming, to obtain subject representation information corresponding to the dialogue text ； wherein is an information transformation network, is a topic switch discrimination network, is topic switch probability information.

Images