End-to-end multi-turn dialogue rewriting method, system and storage medium for integrated dialogue detection

Abstract

This invention relates to the field of natural language processing technology, providing an end-to-end multi-turn dialogue rewriting method, system, and storage medium that integrates dialogue detection. In the proposed end-to-end multi-turn dialogue rewriting method, a joint learning approach is used to train the detector and generator, solving the problems of cascading errors and incoherent sentences after matching and filling in the Pipeline method, thus improving the detector's detection performance. By having the detector ignore sentences that do not need rewriting and integrating dialogue detection information into the generator, the slow speed and repetitive encoding problems of the Seq2Seq method are solved. The dialogue rewriting process of this invention does not require the generator to repeatedly encode the text, improving the model's prediction speed. Simultaneously, integrating dialogue detection information into the generator improves the generation quality of the rewritten sentences.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more particularly to multi-turn dialogue systems, specifically to an end-to-end multi-turn dialogue rewriting method, system, and storage medium that integrates dialogue detection. Background Technology

[0002] Conversational AI is primarily used in industry for human-computer dialogue and question-answering systems, including one-turn dialogue systems and multi-turn dialogue systems. One-turn dialogue systems involve a single question and answer, where other questions and answers have no impact on the current question or answer. Multi-turn dialogue systems, on the other hand, typically complete the dialogue through multiple questions and answers. In multi-turn dialogues, user input often contains omissions or missing references; directly using the content of the current turn to answer can lead to ambiguous responses or an inability to respond.

[0003] For example:

[0004] User: Please check my points.

[0005] "The robot can check your order details, logistics, coupons, and points through the official online store - personal center."

[0006] User: I can't find this option in my user profile.

[0007] If you directly reply to the last sentence, "Open the personal center, there is no such option," it is impossible to determine what "option" it is, so the machine cannot give an accurate answer.

[0008] Therefore, in multi-round dialogues, it is often necessary to modify the content of the current round of dialogue based on historical information, and fill in and complete the information, which is called multi-round dialogue rewriting.

[0009] If the message "Open my personal center, but there's no such option" is rewritten as "Open my personal center, but there's no points option," then the bot can provide a clearer response.

[0010] The commonly used multi-turn dialogue rewriting methods are: (1) Pipeline-based dialogue rewriting method, which first performs rewriting detection on the current dialogue, then extracts the elements to be filled from the historical dialogue, and finally fills the candidate elements according to the rules or matching method to complete the dialogue rewriting; (2) Seq2Seq-based dialogue rewriting method, which directly inputs the historical dialogue and the current dialogue into the end-to-end model, or first extracts the elements to be filled from the historical dialogue, splices them into the current dialogue, and then generates the rewriting content of the current dialogue.

[0011] In practical applications, Pipeline-based methods often suffer from severe cascading errors, resulting in incoherent sentences after matching and filling. Seq2Seq-based methods often require word-by-word generation, leading to significant time consumption and a poor user experience. Summary of the Invention

[0012] The purpose of this invention is to propose an end-to-end multi-turn dialogue rewriting method that integrates dialogue detection. This method integrates relevant information from dialogue detection into the generator and trains the detector and generator through joint learning, thereby improving the final rewriting effect and dialogue rewriting speed.

[0013] According to a first aspect of the present invention, an end-to-end multi-turn dialogue rewriting method incorporating dialogue detection is proposed, comprising the following steps:

[0014] Step 101: Obtain user input from the dialogue system, including the current round of dialogue C and the previous round of dialogue U, where each piece of data in the previous round of dialogue U represents the dialogue text of one end;

[0015] Step 102: For the user input, use the [SEP] symbol to concatenate the dialogue text from each end in sequence, and extract features from the concatenation results to obtain the corresponding feature ID;

[0016] Step 103: Input the obtained feature ID into the detector of the Transformer-based model M to detect the current round of dialogue and obtain the state of each character in the current round of dialogue C. The state labels include: A-insertion, D-deletion, R-replacement and K-invariance.

[0017] Step 104: Based on the state of the character obtained in step 103, determine whether all states are K. If so, proceed to step 105; otherwise, proceed to step 106.

[0018] Step 105: Directly return to the user's current dialogue C without performing any rewriting operations;

[0019] Step 106: Embed the character states obtained in Step 103 into words to achieve vectorization, and then embed the vectorized character states into the representation E. l The concatenation vector E″ is obtained by concatenating the detector output E′ with the concatenation vector E″.

[0020] Step 107: Combine the concatenated vector E″ and the starting symbol <sos>The input is fed into the generator of the Transformer-based model M for word-by-word generation. The generator is a standard Transformer model Decoder structure, and its output is passed through a fully connected layer and activated by the softmax function for word-by-word generation.

[0021] Step 108: When the predicted label is the end symbol <eos>When the time comes, stop generating and output the sentence C′ rewritten from the user's current round of dialogue.

[0022] As an optional implementation, the detector is configured as an Encoder structure of a standard Transformer model, detecting the current round of dialogue and passing its output through a fully connected layer, then using a softmax activation function to obtain the state of each word; its specific implementation process includes:

[0023] First, after embedding the three feature IDs, sum them bit by bit to obtain the vector representation E of the text:

[0024] E = E input (input_id)+E position (position_id)+E state (dialog_state_id)

[0025] Among them, E input (input_id) is the word embedding representation obtained after embedding representation of input_id; E position (position_id) is the position embedding representation obtained after embedding position_id; E state (dialog_state_id) is the dialog state embedding representation obtained after embedding the dialog_state_id;

[0026] Then, the obtained vector representation E is input into the encoder of the standard Transformer model to obtain the vector representation E′ encoded by the encoder:

[0027] E′ = TransformerEncoder(E)

[0028] Where TransformerEncoder is the standard Transformer model encoder, and E′ is the vector representation of E encoded by the encoder.

[0029] Finally, after passing through a fully connected layer and activating it, we obtain the state label for each word:

[0030] p Dlabel = softmax(WE′+b)

[0031] Dlabel′=argmax(p Dlabel )

[0032] Where, p Dlabel represents the probability of predicting each word state, softmax is the activation function, W and b are the weights and biases of the fully connected layer at that point, respectively; Dlabel′ represents the label of the predicted word state, and argmax is the value of p. Dlabel The label with the highest probability value.

[0033] As an optional implementation, in step 107, the splicing vector E″ and the start symbol are... <sos>The input is fed into the generator of the Transformer-based model M for word-by-word generation, including the following operations:

[0034] E g =TransformerDecoder(E″)

[0035] p Glabel = softmax(WE+b)

[0036] Glabel′=argmax(p Glabel )

[0037] Wherein, TransformerDecoder is the standard Transformer model decoder, E g The vector representation E″ is the vector representation decoded by the decoder; p Glabel argmax represents the probability of predicting each character in the vocabulary, softmax is the activation function, W and b are the weights and biases of the fully connected layer at that point, respectively; Glabel′ represents the label of the predicted character, and argmax is the value of p. Glabel The label with the highest probability value.

[0038] As an optional implementation, the Transformer-based model M is pre-trained based on a training sample set, wherein, during each training step, the detector's loss value is... d Loss value of generator g Dynamic weighting is used to achieve joint training of the models.

[0039] As an optional implementation, the detector's loss value Loss d The calculation process is expressed as follows:

[0040]

[0041] Where Dlabel is the true label of the detector text, p Dlabel Predict the probability of each word state for the detector;

[0042] The generator's loss value Loss g The calculation process is expressed as follows:

[0043]

[0044] Where Glabel is the actual label of the generator text, p Glabel Predict the probability of each word in the dictionary for the generator;

[0045] Finally, the losses of the detector and generator are dynamically weighted and fused to determine the final model loss function L. oss :

[0046] Loss = w * Loss d +(1-w)*Loss g

[0047] Where w represents the weight value of the detector's loss, which is a trainable parameter;

[0048] This enables joint training of the detector and the generator.

[0049] In a second aspect of the present invention, a computer system is also provided, comprising:

[0050] One or more processors;

[0051] The memory stores operable instructions that, when executed by one or more processors, implement the aforementioned end-to-end multi-turn dialogue rewriting method for fusion dialogue detection.

[0052] In a third aspect of the present invention, a computer-readable medium for storing a computer program is also provided, the computer program including instructions executable by one or more computers, the instructions, when executed by the one or more computers, implementing the process of the aforementioned end-to-end multi-turn dialogue rewriting method for fusion dialogue detection.

[0053] In conjunction with the embodiments of the present invention described above, existing pipeline-based dialogue rewriting methods often suffer from severe cascading errors, resulting in incoherent sentences after matching and filling. Seq2Seq-based dialogue rewriting methods often require word-by-word generation and rewrite many sentences that do not need rewriting, leading to significant time consumption. Therefore, in the end-to-end multi-turn dialogue rewriting method with integrated dialogue detection proposed in this invention, the detector and generator are trained through joint learning, solving the problems of cascading errors and incoherent sentences after matching and filling in the pipeline method, thus improving the detector's detection performance. By ignoring sentences that do not need rewriting and integrating dialogue detection information into the generator, the slow speed and repetitive encoding problems of the Seq2Seq method are solved. The dialogue rewriting process of this invention does not require the generator to repeatedly encode the text, improving the model's prediction speed. Simultaneously, integrating dialogue detection information into the generator improves the generation effect of rewritten sentences.

[0054] Compared with existing technologies, the significant advantages of the end-to-end multi-turn dialogue rewriting method with fusion dialogue detection of the present invention are as follows:

[0055] 1. In the dialogue state detection part, when the current dialogue state is detected to be that no modification is needed, the output is returned directly without entering the model dialogue generation part, thereby improving the model prediction speed;

[0056] 2. When the detected dialogue needs to be modified, the dialogue generation part of the model uses the detector's encoded output, eliminating the need for the generator to repeatedly encode the text, thereby improving the model's prediction speed.

[0057] 3. In the dialogue generation part of the model, relevant information from dialogue detection is integrated into the generator to guide the dialogue generation at each step, thereby improving the generation effect of rewritten sentences.

[0058] 4. The end-to-end multi-turn dialogue rewriting model that integrates dialogue detection adopts a joint training method for detectors and generators. During the training process, they influence and optimize each other, thereby improving the performance of dialogue detection and dialogue generation.

[0059] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below may be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other. Furthermore, all combinations of the claimed subject matter are considered part of the inventive subject matter of this disclosure.

[0060] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0061] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein:

[0062] Figure 1 This is a flowchart illustrating the end-to-end multi-turn dialogue rewriting method for fusion dialogue detection according to an embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of the model structure based on the Transformer structure in an embodiment of the present invention.

[0064] Figure 3 This is a schematic diagram of the training process of a model based on the Transformer structure according to an embodiment of the present invention. Detailed Implementation

[0065] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0066] Various aspects of the invention are described in this disclosure with reference to the accompanying drawings, which illustrate numerous illustrative embodiments. The embodiments of this disclosure are not necessarily intended to encompass all aspects of the invention. It should be understood that the various concepts and embodiments described above, as well as those described in more detail below, can be implemented in any of many ways, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0067] Combination Figure 1 The exemplary process illustrated here is an end-to-end multi-turn dialogue rewriting method for fusion dialogue detection, which includes the following implementation steps:

[0068] Step 101: Obtain user input from the dialogue system, including the current round of dialogue C and the previous round of dialogue U, where each piece of data in the previous round of dialogue U represents the dialogue text of one end;

[0069] Step 102: For the user input, use the [SEP] symbol to concatenate the dialogue text from each end in sequence, and extract features from the concatenation results to obtain the corresponding feature ID;

[0070] Step 103: Input the obtained feature ID into the detector of the Transformer-based model M to detect the current round of dialogue and obtain the state of each character in the current round of dialogue C. The state labels include: A-insertion, D-deletion, R-replacement and K-invariance.

[0071] Step 104: Based on the state of the character obtained in step 103, determine whether all states are K. If so, proceed to step 105; otherwise, proceed to step 106.

[0072] Step 105: Directly return to the user's current dialogue C without performing any rewriting operations;

[0073] Step 106: Embed the character states obtained in Step 103 into words to achieve vectorization, and then embed the vectorized character states into the representation E. l The concatenation vector E″ is obtained by concatenating the detector output E′ with the concatenation vector E″.

[0074] Step 107: Combine the concatenated vector E″ and the starting symbol <sos>The input is fed into the generator of the Transformer-based model M for word-by-word generation. The generator is a standard Transformer model Decoder structure, and its output is passed through a fully connected layer and activated by the softmax function for word-by-word generation.

[0075] Step 108: When the predicted label is the end symbol <eos>When the time comes, stop generating and output the sentence C′ rewritten from the user's current round of dialogue.

[0076] In step 102, the concatenation of the dialogue text is set to be performed sequentially according to the order of the dialogue, starting from the first round of dialogue and continuing until the end of the current round of dialogue.

[0077] In step 102, the feature ID includes input_id, position_id, and dialog_state_id; input_id is a sequence in which each element is the dictionary ID number corresponding to each character in the text; position_id is a sequence in which elements start from 0 and increment by one, representing the position of each character in the text; dialog_state_id is a sequence in which each element is the sequence number of the dialogue sequence corresponding to each character in the text.

[0078] Combination Figure 2 As shown, in step 103, the detector is configured as an Encoder structure of a standard Transformer model to detect the current round of dialogue, and its output is passed through a fully connected layer and activated by a softmax function to obtain the state of each word. The specific implementation process includes:

[0079] First, after embedding the three feature IDs, sum them bit by bit to obtain the vector representation E of the text:

[0080] E = E input (input_id)+E position (position_id)+E state (dialog_state_id)

[0081] Among them, E input (input_id) is the word embedding representation obtained after embedding representation of input_id; E position (position_id) is the position embedding representation obtained after embedding position_id; E state (dialog_state_id) is the dialog state embedding representation obtained after embedding the dialog_state_id;

[0082] Then, the obtained vector representation E is input into the encoder of the standard Transformer model to obtain the vector representation E′ encoded by the encoder:

[0083] E′ = TransformerEncoder(E)

[0084] Where TransformerEncoder is the standard Transformer model encoder, and E′ is the vector representation of E encoded by the encoder.

[0085] Finally, after passing through a fully connected layer and activating it, we obtain the state label for each word:

[0086] p Dlabel = softmax(WE′+b)

[0087] Dlabel′=argmax(p Dlabel )

[0088] Where, p Dlabel represents the probability of predicting each word state, softmax is the activation function, W and b are the weights and biases of the fully connected layer at that point, respectively; Dlabel′ represents the label of the predicted word state, and argmax is the value of p. Dlabel The label with the highest probability value.

[0089] As an optional embodiment, in step 106, the character states obtained in step 103 are word-embedded to achieve vectorization, and then the vectorized character states are embedded into the representation E. l The concatenation vector E″ is obtained by concatenating the concatenation vector E″ with the detector output E′, including the following operations:

[0090] E l =E Dlabel (Dlabel′)

[0091] E″=E l +E′

[0092] Where Dlabel′ is the result of the detector of model M predicting the state of each character in step 103, and E Dlabel E is a character state embedding representation matrix. l E″ is the character state embedding representation obtained after embedding representation of Dlabel′; E″ is the bitwise summation result of the TransformerEncoder output of model M and the character state embedding representation.

[0093] As an optional embodiment, in step 107, the splicing vector E″ and the start symbol are... <sos>The input is fed into the generator of the Transformer-based model M for word-by-word generation, including the following operations:

[0094] E g =TransformerDecoder(E″)

[0095] p Glabel = softmax(WE+b)

[0096] Glabel′=argmax(p Glabel )

[0097] Wherein, TransformerDecoder is the standard Transformer model decoder, E g The vector representation E″ is the vector representation decoded by the decoder; p Glabel argmax represents the probability of predicting each character in the vocabulary, softmax is the activation function, W and b are the weights and biases of the fully connected layer at that point, respectively; Glabel′ represents the label of the predicted character, and argmax is the value of p. Glabel The label with the highest probability value.

[0098] As an optional embodiment, the Transformer-based model M is pre-trained based on a training sample set, wherein, during each training step, the detector's loss value is... d Loss value of generator g Dynamic weighting is used to achieve joint training of the models.

[0099] Among them, the detector's loss value is Loss d The calculation process is expressed as follows:

[0100]

[0101] Where Dlabel is the true label of the detector text, p Dlabel Predict the probability of each word state for the detector;

[0102] The generator's loss value Loss g The calculation process is expressed as follows:

[0103]

[0104] Where Glabel is the actual label of the generator text, p Glabel Predict the probability of each word in the dictionary for the generator;

[0105] Finally, the losses from the detector and generator are dynamically weighted and fused to determine the final model loss function, Loss:

[0106] Loss = w * Loss d +(1-w)*Loss g

[0107] Where w represents the weight value of the detector's loss, which is a trainable parameter;

[0108] This enables joint training of the detector and the generator.

[0109] Therefore, the end-to-end multi-turn dialogue rewriting method proposed in this invention integrates dialogue detection information into the generator and trains the detector and generator through joint learning, thereby improving the final rewriting effect and dialogue rewriting speed. By jointly learning the detector and generator, it solves the problems of cascading errors and incoherent sentences after matching and filling in existing pipeline methods, and improves the detection effect of the detector. Furthermore, by having the detector ignore sentences that do not need rewriting and integrating dialogue detection information into the generator, it solves the problems of slow speed and repetitive encoding in the Seq2Seq method.

[0110] Below, we will describe in more detail an exemplary implementation of the above process with reference to specific embodiments.

[0111] Step 101: Obtain user input from the dialogue system, including the current round of dialogue C and the previous rounds of dialogue U, U = {U1, U2, ..., Un};

[0112] The following is an example dialogue:

[0113] User: Please check my points.

[0114] "The robot can check your order details, logistics, coupons, and points through the official online store - personal center."

[0115] User: "I opened my personal center, but there's no such option."

[0116] but:

[0117] In the current dialogue, option C = Open Personal Center, but this option is not available.

[0118] Historical Dialogue:

[0119] U1 = Check the integral;

[0120] U2 = You can check your order details, logistics and delivery, coupons and points through the official store - personal center.

[0121] Step 102: Use the "[SEP]" symbol to concatenate the dialogue text from the input obtained in step 101.

[0122] Taking the aforementioned example dialogue as an example, the concatenation result is as follows:

[0123] "To check your points [SEP], you can go to the **Official Store - Personal Center: You can check your order details, logistics and delivery, coupons and points [SEP]. But I don't have this option in the Personal Center."

[0124] Then, the concatenation result is converted to an ID and the corresponding feature IDs are extracted, including: input_id, position_id and dialog_state_id.

[0125] input_id is a sequence, where each element is the dictionary ID number corresponding to each character in the text.

[0126] position_id is a sequence, with elements starting from 0 and incrementing by one each time, representing the position of each character in the text.

[0127] dialog_state_id is a sequence, where each element is the sequence number of the dialogue sequence corresponding to each word in the text.

[0128] Step 103: Input the three feature IDs obtained in step 102 into the detector of model M to detect the current round of dialogue and obtain the state of each character in the current round of dialogue C. The state labels include: A (insertion), D (deletion), R (replacement) and K (unchanged).

[0129] like Figure 2 The schematic diagram of model M shows that the detector is configured as the Encoder structure of a standard Transformer model, and its output is passed through a fully connected layer and activated by the softmax function to obtain the label state of each word.

[0130] As an example, the feature ID is input into the detector, and the specific operations are as follows:

[0131] First, after embedding the three feature IDs, they are summed bit by bit to obtain the vector representation of the text:

[0132] E = E input (input_id)+E position (position_id)+E state (dialog_state_id)

[0133] Among them, E input (input_id) is the word embedding representation obtained after embedding representation of input_id; E position (position_id) is the position embedding representation obtained after embedding position_id; E state (dialog_state_id) is the dialog state embedding representation obtained after embedding the dialog_state_id;

[0134] Then, the obtained vector representation E is input into the standard Transformer model Encoder to obtain the state E′ of each character:

[0135] E′ = TransformerEncoder(E)

[0136] Where TransformerEncoder is the standard Transformer model encoder, and E′ is the vector representation of E encoded by the encoder.

[0137] Finally, after passing through a fully connected layer and activating it, the label state of each word is obtained:

[0138] p Dlabel = softmax(WE′+b)

[0139] Dlabel′=argmax(p Dlabel )

[0140] Where, p Dlabel represents the probability of predicting each word state, softmax is the activation function, W and b are the weights and biases of the fully connected layer, respectively; Dlabel′ represents the label of the predicted word state, and argmax is the value of p. Dlabel The label with the highest probability value.

[0141] Step 104: Based on the character status obtained in Step 103, determine whether all statuses are K. If yes, proceed to Step 105; otherwise, proceed to Step 106.

[0142] Step 105: Directly return to the user's current dialogue C; no rewriting operation is required at this point.

[0143] Step 106: Embed the character states obtained in Step 103 into words to achieve vectorization, and concatenate them with the detector output E′ to obtain the concatenated vector E″. The specific operation process includes:

[0144] E l =E Dlabel (Dlabel′)

[0145] E″=E l +E′

[0146] Where Dlabel′ is the predicted state result of model M detector for each character in step 103, and E Dlabel E is a character state embedding representation matrix. l E″ is the character state embedding representation obtained after embedding representation of Dlabel′; E″ is the bitwise summation result of the output of the M detector TransformerEncoder and the character state embedding representation.

[0147] Step 107: Combine the concatenation vector E″ and the starting symbol " <sos>The input is fed into the generator of model M for word-by-word generation. The generator is a standard Transformer model Decoder structure, and its output is passed through a fully connected layer and activated by the softmax function for word-by-word generation, as follows:

[0148] E g =TransformerDecoder(E″)

[0149] p Glabel = softmax(WE+b)

[0150] Glabel′=argmax(p Glabel )

[0151] Wherein, TransformerDecoder is the standard Transformer model decoder, E g The vector representation E″ is the vector representation decoded by the decoder; p Glabel argmax represents the probability of predicting each character in the vocabulary, softmax is the activation function, W and b are the weights and biases of the fully connected layer, respectively; Glabel′ represents the label of the predicted character, and argmax is the value of p. Glabel The label with the highest probability value.

[0152] Step 108: When the predicted label is the end symbol " <eos>When the time comes, stop generating and output the sentence C′ rewritten from the user's current round of dialogue.

[0153] Thus, the process of rewriting the dialogue is completed.

[0154] Combination Figure 3 The process shown exemplifies the training of a Transformer-based model M and the process of saving the optimal model.

[0155] Step 201: Crawl, clean, and label the multi-turn dialogue dataset to construct dataset D.

[0156] Step 202: Split the dataset in an 8:1:1 ratio to construct the training set D. train Validation set D dev and test set D test .

[0157] Step 203: Train the multi-turn dialogue rewriting model, and test it on the validation set D after a predetermined number of training steps (e.g., 1000 steps). dev Perform a verification above.

[0158] During each training step, the loss value of the detector is... d Loss value of generator g Dynamic weighting is used to achieve joint training of the models. As follows:

[0159] First, calculate the detector's loss value. d :

[0160]

[0161] Where Dlabel is the true label of the detector text, p blabel Predict the probability of each word state for the detector.

[0162] Secondly, calculate the generator's loss value. g :

[0163]

[0164] Where Glabel is the actual label of the generator text, p Glabel Predict the probability of each character in the dictionary for the generator.

[0165] Finally, the detector and generator are dynamically fused:

[0166] Loss = w * Loss d +(1-w)*Loss g

[0167] Where w is a trainable parameter, enabling joint training of the detector and generator.

[0168] Step 204: Determine whether the current verification result is better than the previous result. If so, proceed to step 205; otherwise, proceed to step 206.

[0169] Step 205: Save the current model M and reset the early stopping number n to 0.

[0170] Step 206: Stop adding 1 to the number n as early as possible.

[0171] Step 207: Determine if the early stopping number n is greater than 5. If yes, proceed to step 208; otherwise, proceed to step 203.

[0172] Step 208: Stop training.

[0173] Based on the implementation of the end-to-end multi-turn dialogue rewriting method for fusion dialogue detection in the above embodiments, a computer system is also proposed according to the embodiments disclosed in this invention, including: one or more processors and memory.

[0174] The aforementioned memory is configured to store executable instructions, which, when executed by one or more processors, implement the aforementioned end-to-end multi-turn dialogue rewriting method for fusion dialogue detection.

[0175] In conjunction with the implementation of the end-to-end multi-turn dialogue rewriting method for fusion dialogue detection in the above embodiments, the present invention also proposes a computer-readable medium for storing a computer program, the computer program including instructions executable by one or more computers, the instructions implementing the aforementioned end-to-end multi-turn dialogue rewriting method for fusion dialogue detection when executed by the one or more computers.

[0176] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.< / eos> < / sos> < / sos> < / eos> < / sos> < / sos> < / eos> < / sos>

Claims

1. An end-to-end multi-turn dialogue rewriting method incorporating dialogue detection, characterized in that, Includes the following steps: Step 101: Obtain user input from the dialogue system, including the current round of dialogue C and the previous round of dialogue U, where each piece of data in the previous round of dialogue U represents the dialogue text of one end; Step 102: For the user input, use the [SEP] symbol to concatenate the dialogue text from each end in sequence, and extract features from the concatenation results to obtain the corresponding feature ID; Step 103: Input the obtained feature ID into the detector of the Transformer-based model M to detect the current round of dialogue and obtain the state of each character in the current round of dialogue C. The state labels include: A-insertion, D-deletion, R-replacement and K-invariance. Step 104: Based on the state of the character obtained in step 103, determine whether all states are K. If so, proceed to step 105; otherwise, proceed to step 106. Step 105: When all the state labels are unchanged labels K, the original current round dialogue is directly output as the rewriting result, and no rewriting operation is performed. Step 106: When the state labels are not all invariant labels K, the character states obtained in step 103 are word-embedded to achieve vectorization, and then the vectorized character states are embedded into the representation E. l The concatenation vector E'' is obtained by concatenating the detector output E' with the concatenation vector E''. Step 107: Combine the concatenated vector E'' and the starting symbol <sos> The input is fed into the generator of the Transformer-based model M for character-by-character generation. The generator is a standard Transformer model Decoder structure, and its output is passed through a fully connected layer and a softmax activation function to generate characters character by character, predicting the labels of the generated characters until the end symbol is predicted.< / sos> Step 108: When the predicted label is the end symbol <eos> When the time comes, stop generating and output the sentence C', which is the rewritten version of the user's current dialogue.< / eos> The model M is jointly trained by a weighted sum of the detector loss and the generator loss.

2. The end-to-end multi-turn dialogue rewriting method with fusion dialogue detection according to claim 1, characterized in that, In step 102, the concatenation of the dialogue text is set to be performed sequentially according to the order of the dialogue, starting from the first round of dialogue and continuing until the end of the current round of dialogue.

3. The end-to-end multi-turn dialogue rewriting method with fusion dialogue detection according to claim 1, characterized in that, In step 102, the feature ID includes input_id, position_id, and dialog_state_id; The input_id is a sequence, where each element is the dictionary ID number corresponding to each character in the text; The position_id is a sequence in which the elements start from 0 and increment by one, representing the position of each character in the text. The dialog_state_id is a sequence, where each element is the sequence number of the dialogue sequence corresponding to each word in the text.

4. The end-to-end multi-turn dialogue rewriting method with fusion dialogue detection according to claim 1, characterized in that, In step 103, the detector is set up as an Encoder structure of a standard Transformer model to detect the current round of dialogue, and its output is passed through a fully connected layer and the state of each word is obtained through the softmax activation function. Its specific implementation process includes: First, after embedding the three feature IDs, sum them bit by bit to obtain the vector representation E of the text: ； The word embedding representation obtained after embedding representation of input_id; The position embedding representation obtained after embedding representation of position_id; The dialog state embedding representation obtained after embedding representation of dialog_state_id; Then, the obtained vector representation E is input into the encoder of the standard Transformer model to obtain the vector representation E encoded by the encoder: ； For the standard Transformer model Encoder, E' is the vector representation of E after being encoded by the encoder; Finally, after passing through a fully connected layer and activating it, we obtain the state label for each word: ； ； This indicates the probability of predicting the state of each word; For activation functions; W and b are the weights and biases of the fully connected layer at that point, respectively. A label representing the predicted state of each word; To obtain The label with the highest probability value.

5. The end-to-end multi-turn dialogue rewriting method with fusion dialogue detection according to claim 4, characterized in that, In step 106, the character states obtained in step 103 are word-embedded to achieve vectorization, and then the vectorized character states are embedded into the representation. With detector output To concatenate the vectors and obtain the concatenated vector E'', the following operations are performed: ； ； in, This refers to the result of the detector in model M predicting the state for each character in step 103. Embed the representation matrix for the character state. for The character state embedding representation obtained after embedding representation; To make the detector of model M The output is the bitwise sum of the character state embedding representation.

6. The end-to-end multi-turn dialogue rewriting method with fusion dialogue detection according to claim 5, characterized in that, In step 107, the spliced ​​vectors are... and the starting symbol <sos> The input is fed into the generator of the Transformer-based model M for word-by-word generation, including the following operations:< / sos> ； ； ； in, For the standard Transformer model Decoder, Vector representation Vector representation after decoding; This represents the probability of predicting each character in the vocabulary. For activation function, and These are the weights and biases of the fully connected layer at that location; Labels indicating predicted words, To obtain The label with the highest probability value.

7. The end-to-end multi-turn dialogue rewriting method for fusion dialogue detection according to any one of claims 1-6, characterized in that, The Transformer-based model M is pre-trained using a training sample set, wherein, at each training step, the detector's loss value is... Loss value of generator Dynamic weighting is used to achieve joint training of the models.

8. The end-to-end multi-turn dialogue rewriting method with fusion dialogue detection according to claim 7, characterized in that, The loss value of the detector The calculation process is expressed as follows: ； in, The true label of the detector text. Predict the probability of each word state for the detector; The loss value of the generator The calculation process is expressed as follows: ； in, The actual tags for the generator text. Predict the probability of each word in the dictionary for the generator; Finally, the losses from the detector and generator are dynamically weighted and fused to determine the final model loss function, Loss: ； in, The weights representing the loss of the detector are trainable parameters; This enables joint training of the detector and the generator.

9. A computer system, characterized in that, include: One or more processors; A memory that stores operable instructions, said instructions being executed by one or more processors in the process of performing the end-to-end multi-turn dialogue rewriting method for fusion dialogue detection as described in any one of claims 1 to 8.

10. A computer-readable medium for storing computer programs, characterized in that, The computer program includes instructions executable by one or more computers, which, when executed by the one or more computers, implement the process of the end-to-end multi-turn dialogue rewriting method for fusion dialogue detection as described in any one of claims 1 to 8.

Images