Human-computer multi-turn dialogue rewriting method based on Transformer pointer extraction

Abstract

The present invention discloses a method for rewriting human-computer multi-round dialogues based on Transformer pointer extraction, the implementation steps of which are: constructing a text semantic relevance recognition network and a semantically missing text rewriting network; generating a training set; training the text semantic relevance recognition network and the semantically missing text rewriting network; determining whether the semantics of user input texts are related; and rewriting semantically missing texts. The present invention utilizes the Transformer pre-trained model for feature extraction and encoding, and utilizes the technical approach of extracting key information content of the text through pointer addresses to rewrite the user text. This allows the present invention to determine whether the user's current input text needs to be rewritten, and the rewriting of the user's semantically missing text is high-quality and time-saving. It can be used for rewriting semantically missing texts in the field of human-computer multi-round dialogues.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and more specifically, to a method for rewriting multi-turn human-computer dialogues based on Transformer pointer extraction in the field of natural language processing. By rewriting human-computer dialogue text, this invention provides the necessary technical support for machines in multi-turn dialogues to understand the semantics of user pronoun references and informational default text. Background Art

[0002] In the field of human-computer dialogue, there are two types of dialogue, single-turn and multi-turn, depending on the number of rounds of interaction. Single-turn dialogue technology is relatively mature, including search-based and generative methods. However, these generative methods are very limited in their ability to interpret user expressions, and their application scenarios are limited. In multi-turn dialogue scenarios, users often use semantically missing expressions, such as pronoun references and missing information. This leads to unclear references and missing sentence components, making it difficult for machines to understand the semantics of these expressions, ultimately resulting in poor conversation quality. Rewriting in multi-turn dialogue involves rewriting the semantically missing text of the user's current input, combining it with the previous round of input, restoring the reference or missing information, and helping the machine understand the user's true intentions.

[0003] NetEase (Hangzhou) Network Co., Ltd. proposed a multi-round dialogue rewriting method in its patent application document "A Multi-Round Dialogue Rewriting Method and Device" (Patent Application Number: 202110010037.2, Application Publication Number: CN 112597777 A). This method uses a sequence labeling algorithm to label the user's text to be rewritten and obtains the sequence label corresponding to each sequence in the text, which can effectively improve the quality of the network's rewriting of the user's text. The Transformer stacking structure is used to complete sequence labeling and decoding generation, which can load the weights of the pre-trained model. Through the autoregressive algorithm, the content corresponding to each sequence label is parsed, which can effectively improve the speed of the network rewriting the user's text. However, the method still has the disadvantage that it cannot determine whether the user's current input text is a semantically missing text with pronoun reference and missing information, and therefore cannot determine whether the user's current input text needs to be rewritten.

[0004] Nanjing University of Science and Technology proposed a knowledge-driven dialogue rewriting method in its patent application, "A Knowledge-Driven Dialogue Rewriting Method and System" (Patent Application No.: 202111251522.5, Publication No.: CN 114036271 A). This method leverages the Transformer's attention mechanism to capture dependencies between words in text, addressing the network's limited ability to focus on the deeper semantics of the text and extracting semantic features from it, as is the case with existing dialogue rewriting techniques. This approach introduces the idea of ​​generating rewritten text by copying the user's conversation history using pointer addresses, simplifying the multi-turn dialogue problem to a single-turn one and addressing the difficulty machines have in understanding the true semantics of text where user semantics are missing. However, this method still has drawbacks: it uses six Transformer layers as both encoder and decoder, preventing the network from loading some pre-trained model weights. Furthermore, the proposed pointer network-based approach to generating rewritten text requires generating the rewritten text from scratch, making the network's rewriting process very time-consuming. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a method for rewriting human-computer multi-round dialogues based on Transformer pointer extraction, so as to solve the problems that the existing methods for rewriting human-computer multi-round dialogues cannot determine whether the user's current input text needs to be rewritten, and the rewriting quality of the user's semantically missing text is low and time-consuming.

[0006] The technical idea for achieving the purpose of the present invention is that, in the process of constructing a text semantic relevance recognition network, the present invention utilizes the BERT pre-trained model of Transformer to extract and encode text features, and then focuses on the deep semantics of the text, and uses a fully connected neural network to highly purify the semantic features of the text, solving the technical problem that the machine cannot judge whether the user's current input text needs to be rewritten. In the process of constructing a semantically missing text rewriting network, the present invention utilizes the rbt3 pre-trained model of Transformer to extract and encode text features, extracts the pointer address of the key information of the text, including the keyword position start pointer, the keyword position end pointer, the default position pointer, the pronoun position start pointer and the pronoun position end pointer, and extracts the keyword according to the pointer address to replace the pronoun of the text to be rewritten or insert it into the default position, thereby obtaining a rewritten complete semantic text, solving the technical problem of low quality and long time consumption of the network rewriting user semantically missing text.

[0007] The implementation steps of the present invention are as follows:

[0008] Step 1: Build a text semantic relevance recognition network:

[0009] A four-layer text semantic relevance recognition network was constructed, with the following structure connected in series: data processing layer, semantic encoding layer, classification processing layer, and output layer. The semantic encoding layer was implemented using the BERT pre-training model. The word embedding vector dimension was set to 768, and the maximum allowed length of the text was set to 10. The classification processing layer consisted of two identical first classification units and a second classification unit in series. Each classification unit consisted of a Dropout module and a fully connected module in series. The Dropout ratio was set to 0.2, the output dimension of the fully connected module in the first classification unit was set to 8, and the output dimension of the fully connected module in the second classification unit was set to 2. The output layer consisted of Softmax units, and the output values ​​were all set to 0 or 1.

[0010] Step 2: Build a semantic missing text rewriting network:

[0011] A four-layer semantically missing text rewriting network was constructed, with a structure consisting of a data processing layer, a semantic encoding layer, a pointer prediction layer, and an output layer. The semantic encoding layer was implemented using the Transformer RBT3 pre-trained model, with the word embedding vector dimension set to 768 and the maximum allowed text length set to 50. The pointer prediction layer was implemented using a pointer extraction method, which extracted the pointer addresses of key information in the text and used them to replace the referents in the text to be rewritten or insert them into default locations to obtain the rewritten complete semantic text. The training sample batch size was set to 16, and the total training round size was set to 20. The output layer was implemented using a decoder, and the network's rewriting evaluation metric for the sample data was set to the Rouge-1 value.

[0012] Step 3: Generate training set:

[0013] Step 3.1: Combine the two texts and their corresponding semantic relevance labels into a set of text semantic relevance recognition data in the format of (label, q1, q2), where q1 and q2 represent the two texts used for semantic relevance recognition, and label represents a label with a value of 0 or 1, corresponding to whether the two texts are semantically unrelated or related, respectively. Using the same method, select at least 80,000 sets of text semantic relevance recognition data to form a text semantic relevance recognition training set.

[0014] Step 3.2: Group the four texts into a set of data in the format of (a, b, current, right), where a represents the user's input text in the previous round, b represents the system's reply text in the previous round, current represents the text to be rewritten, and right represents the theoretically correct rewritten text. Using the same method, select at least 17,000 sets of data to form the semantically missing text rewriting training set.

[0015] Step 4: Train the text semantic relevance recognition network:

[0016] The text semantic relevance recognition training set is input into the text semantic relevance recognition network, and the predicted labels of the training samples are output. The cross entropy loss function is used to calculate the loss value between the predicted labels and the true labels of the training samples. All parameters in the text semantic relevance recognition network are iteratively updated through the gradient descent method until the loss value no longer changes, thereby obtaining a trained text semantic relevance recognition network.

[0017] Step 5: Train the semantic missing text rewriting network:

[0018] Input the semantic missing text rewriting training set into the semantic missing text rewriting network, output the rewritten complete semantic text, and calculate the Rouge-1 value between the network-rewritten text and the theoretically correct rewritten text. Using the gradient descent method, iteratively update all parameters in the semantic missing text rewriting network until the Rouge-1 value no longer changes. Record all parameters of the training round with the highest Rouge-1 value to obtain the trained semantic missing text rewriting network.

[0019] Step 6: Determine whether the user input text is semantically related:

[0020] The user's previous input text and the user's current input text are input into the trained text semantic relevance recognition network, and the semantic relevance prediction label of the two rounds of user conversation text is output. If the predicted label value is 0, it means that the user's current input text has no semantic relevance with the previous round of input text and no rewriting is required. If the predicted label value is 1, it means that the user's current input text is semantically related to the previous round of input text and rewriting is required to understand the semantics of the current input text, and then execute step 7;

[0021] Step 7: Rewrite the semantically missing text:

[0022] The user's previous input text, the user's current input text, and the machine's previous reply text are all input into the trained semantically missing text rewriting network, and the complete semantic text after rewriting the user's current input text is output. The semantics of the user's current input text is understood based on the rewritten complete semantic text.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] First, since the present invention constructs a text semantic relevance recognition network, it can identify whether the semantics between the two rounds of user input text are related, judge whether the user's current input text needs to be combined with the previous round of text for semantic understanding, and then judge whether the user's current input text needs to be rewritten, which overcomes the defect of the existing technology that cannot judge whether the user's current input text needs to be rewritten, so that the present invention has the advantage of being able to judge whether the user's current input text needs to be rewritten.

[0025] Second, since the present invention constructs a semantically missing text rewriting network, adopts the Transformer's rbt3 pre-training model to process and extract text features, and uses a method based on pointer addresses to extract key information content of the text, it overcomes the defects of the existing technology of low quality and long time consumption in rewriting user texts, making the present invention have the advantages of high quality and short time consumption in rewriting user semantically missing texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 is a simulation diagram of the present invention; wherein, Figure 2(a) is a curve diagram showing the relationship between the accuracy rate and the training rounds obtained by recognizing the text semantic relevance recognition training set by the semantic relevance recognition network and the prior art in simulation experiment 1 of the present invention, Figure 2(b) is a curve diagram showing the relationship between the loss value between the predicted label and the true label and the training rounds obtained by recognizing the text semantic relevance recognition training set by the semantic relevance recognition network and the prior art in simulation experiment 1 of the present invention, Figure 2(c) is a curve diagram showing the relationship between the Rouge-1 value and the training rounds obtained by rewriting the semantic missing text rewriting dataset by the semantic missing text rewriting network and the prior art in simulation experiment 2 of the present invention, and Figure 2(d) is a curve diagram showing the relationship between the consumption time and the training rounds obtained by rewriting the semantic missing text rewriting dataset by the semantic missing text rewriting network and the prior art in simulation experiment 2 of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] Reference Figure 1 , further describing the implementation steps of the embodiment of the present invention.

[0030] Step 1: Build a text semantic relevance recognition network.

[0031] A four-layer text semantic relevance recognition network is built, and its structure is connected in sequence: data processing layer, semantic encoding layer, classification processing layer, and output layer; among them, the semantic encoding layer is implemented by the BERT pre-trained model of Transformer, the word embedding vector dimension is set to 768, and the maximum allowed length of the text is set to 10; the classification processing layer is composed of two first classification units and second classification units with the same structure connected in series, each classification unit is composed of a Dropout module and a fully connected module connected in series, the Dropout ratio is set to 0.2, the output dimension of the fully connected module in the first classification unit is set to 8, and the output dimension of the fully connected module in the second classification unit is set to 2; the output layer is composed of Softmax units, and the output values ​​are all set to 0 or 1.

[0032] Step 2: Build a semantically missing text rewriting network.

[0033] A four-layer semantically missing text rewriting network was constructed, consisting of a data processing layer, a semantic encoding layer, a pointer prediction layer, and an output layer. The semantic encoding layer was implemented using the Transformer RBT3 pre-trained model, with the word embedding vector dimension set to 768 and the maximum allowed text length set to 50. The pointer prediction layer was implemented using a pointer extraction method, extracting the pointer addresses of key text information and replacing the referents in the rewritten text with the keywords extracted from the pointer addresses or inserting them into the default positions to obtain the rewritten complete semantic text. The training sample batch size was set to 16, and the total training round size was set to 20. The output layer was implemented using a decoder, and the network's rewriting evaluation metric for the sample data was set to the Rouge-1 value.

[0034] The pointer addresses include a keyword position start pointer, a keyword position end pointer, a default position pointer, a pronoun position start pointer and a pronoun position end pointer.

[0035] The pointer extraction method is to obtain keywords, pronouns, and default positions by comparing the differences between the current text to be rewritten and the theoretically correctly rewritten text in the training set, and extract the start pointer and end pointer of the keyword position in the user's previous round of input text based on the keywords, extract the start pointer and end pointer of the pronoun position in the current text to be rewritten based on the pronouns, and extract the default position pointer in the current text to be rewritten based on the default position.

[0036] Step 3: Generate a training set.

[0037] Step 3.1, combine the two texts and the corresponding semantic relevance labels into a set of text semantic relevance recognition data in the format of (label, q1, q2), where q1 and q2 respectively represent the two texts used for semantic relevance recognition, label represents the label, and the value is 0 or 1, corresponding to the two texts being semantically irrelevant or relevant, respectively; using the same method, select at least 80,000 sets of text semantic relevance recognition data to form a text semantic relevance recognition training set.

[0038] In step 3.2, the four paragraphs of text are organized into a set of data in the format of (a, b, current, right), where a represents the user's input text in the previous round, b represents the system's reply text in the previous round, current represents the text to be rewritten, and right represents the text after theoretically correct rewriting; using the same method, at least 17,000 sets of data are selected to form a semantically missing text rewriting training set.

[0039] Step 4: Train the text semantic relevance recognition network.

[0040] The text semantic relevance recognition training set is input into the text semantic relevance recognition network. The data processing layer is used to splice the two texts used for semantic relevance recognition to obtain a complete spliced ​​text. The spliced ​​text is then input into the semantic encoding layer. The Transformer's BERT pre-trained model is used to extract features from the spliced ​​text and encode it to obtain a text vector. The text vector is then input into the classification processing layer to obtain a highly purified feature vector. The purified feature vector is then input into the output layer to obtain the predicted label of the semantic relevance between the data texts. The cross-entropy loss function is used to calculate the loss value between the predicted label and the true label of the training sample. Through the gradient descent method, all parameters in the text semantic relevance recognition network are iteratively updated until the loss value no longer changes, thereby obtaining a trained text semantic relevance recognition network.

[0041] According to the following formula, the accuracy of the text semantic relevance recognition network for sample recognition is calculated:

[0042]

[0043] Among them, ACC represents the accuracy of the text semantic relevance recognition network for sample recognition, acc represents the network's predicted label value for the i-th group of training sample data, n represents the total amount of data in the training sample, and ∑ represents the sum.

[0044] The loss between the predicted label and the true label of the training sample is calculated according to the following formula:

[0045]

[0046] Among them, Loss represents the loss value between the predicted label and the true label of the training sample, y i Represents the network's predicted label value for the i-th group of data, label represents the true label value of the i-th group of data, criterion represents the cross entropy function, n represents the total number of data in the i-th group of training samples, ∑ represents the summation operation, and size represents the size of the training sample batch.

[0047] Step 5: Train the semantic missing text rewriting network.

[0048] The semantic missing text rewriting training set is input into the semantic missing text rewriting network, and the data processing layer is used to splice the relevant texts for rewriting to obtain a complete spliced ​​text. The spliced ​​text is then input into the semantic encoding layer, and the Transformer's rbt3 pre-trained model is used to extract features from the spliced ​​text and encode it to obtain a text vector. The text vector is then input into the pointer prediction layer to obtain the pointer address of the key information of the data text, and the pointer address is then input into the output layer to use the decoder to obtain the complete semantic text after the network rewrite. The Rouge-1 value between the text rewritten by the network and the theoretically correct rewritten text is calculated. All parameters in the semantic missing text rewriting network are iteratively updated through the gradient descent method until the Rouge-1 value no longer changes. All parameters of the training round with the highest Rouge-1 value are recorded to obtain a trained semantic missing text rewriting network.

[0049] According to the following formula, the Rouge-1 value of the semantic missing text rewriting network for the sample rewriting is calculated:

[0050]

[0051] Among them, rouge represents the Rouge-1 value of the semantic missing text rewriting network rewriting the i-th group of data, y i Represents the words in the text after the network rewrites the i-th group of data, represents the words in the text that are correctly rewritten by the i-th group of data, and count represents the statistics y i and The same number, len(gram) represents the length of the text that can be theoretically correctly rewritten for the i-th group of data.

[0052] Step 6: Determine whether the user input texts are semantically related.

[0053] The user's previous input text and the user's current input text are input into the trained text semantic relevance recognition network, and the semantic relevance prediction label of the user's two rounds of dialogue text is output. If the predicted label value is 0, it means that the user's current input text has no semantic relevance with the previous round of input text and no rewriting is required; if the predicted label value is 1, it means that the user's current input text is semantically related to the previous round of input text and rewriting is required to understand the semantics of the current input text, and then execute step 7.

[0054] Step 7: Rewrite the semantically missing text.

[0055] The user's previous input, the machine's previous reply, and the user's current input are all fed into a trained semantically missing text rewriting network. The network then outputs a complete semantic rewrite of the user's current input. This complete semantic rewrite is then used to understand the semantics of the user's current input. The machine's previous reply refers to the text it sent after searching for an answer to the user's previous input. This represents the conversation history between the user and the machine and is stored in the Redis third-party middleware.

[0056] The effects of the present invention will be further described below in conjunction with simulation experiments.

[0057] 1. Simulation experiment conditions:

[0058] The hardware platform of the simulation experiment of the present invention is: the processor is Intel i7 10750H CPU, the main frequency is 2.60GHz, and the memory is 32GB.

[0059] The software platform for the simulation experiment of the present invention is: Windows 10 operating system and Python 3.7.

[0060] 2. Simulation content and results analysis:

[0061] There are two simulation experiments of the present invention:

[0062] Simulation experiment 1 uses the text semantic relevance recognition network proposed in the present invention and the existing technology (text semantic relevance recognition method based on bidirectional long short-term memory network) to perform semantic relevance recognition on the input text semantic relevance recognition training set, respectively, to obtain the recognition accuracy and loss value, and then plot the relationship between the accuracy and the training round into a curve as shown in Figure 2(a), and the relationship between the loss value and the training round is plotted into a curve as shown in Figure 2(b).

[0063] In simulation experiment 1, the existing text semantic relevance recognition method based on bidirectional long short-term memory network refers to the method for identifying text semantic relevance based on bidirectional long short-term memory network proposed by Guo Qingsong in "Research on Calculation of Concept and Short Text Semantic Relevance [D]. Guangxi Normal University, 2020.DOI:10.27036 / d.cnki.ggxsu.2020.002023.", referred to as C-BiLSTM.

[0064] Simulation experiment 2 uses the semantically missing text rewriting network proposed in this invention and the existing technology (a method for generating rewritten text based on pointers built on Transformer) to rewrite the input semantically missing text rewriting dataset, respectively, to obtain the Rouge-1 value and consumption time of the rewriting, and then plots the relationship between the Rouge-1 value and the training round into a curve as shown in Figure 2(c), and the relationship between the consumption time and the training round into a curve as shown in Figure 2(d).

[0065] In simulation experiment 2, the existing method of generating rewritten text from pointers built based on Transformer refers to the method of generating rewritten text from pointers built based on Transformer proposed by Hui Su et al. in their published paper "Improving Multi-turn Dialogue Modelling with Utterance ReWriter, CoRR.2020.abs / 1906.07004.", referred to as T-Ptr-λ.

[0066] The effect of the present invention will be further described below in conjunction with the simulation diagram of FIG2.

[0067] The horizontal axis in Figure 2(a) represents the number of training rounds, totaling 20 rounds, and the vertical axis represents the recognition accuracy. The curve marked with a "+" represents the relationship between the accuracy and training rounds obtained using the text semantic relevance recognition network proposed in this invention, while the curve marked with an "*" represents the relationship between the accuracy and training rounds obtained using the prior art C-BiLSTM.

[0068] As can be seen from Figure 2(a), with the increase of training rounds, the accuracy of the text semantic relevance recognition network proposed in the present invention is as high as 0.97, and the highest accuracy of the prior art C-BiLSTM is 0.90, which proves that the text semantic relevance recognition network proposed in the present invention is superior to the prior art in the accuracy of text semantic relevance recognition.

[0069] The horizontal axis in Figure 2(b) represents the number of training rounds, totaling 20 rounds, and the vertical axis represents the loss value. The curve marked with a "+" represents the relationship between the loss value and the number of training rounds obtained using the text semantic relevance recognition network proposed in this invention, while the curve marked with an "*" represents the relationship between the loss value and the number of training rounds obtained using the prior art C-BiLSTM.

[0070] As can be seen from Figure 2(b), with the increase of training rounds, the text semantic relevance recognition network proposed in the present invention has a minimum loss value of 0.10 in the predicted label and the true label of the text semantic relevance recognition training set, while the existing technology C-BiLSTM has a minimum loss value of 0.15, which proves that the text semantic relevance recognition network proposed in the present invention is superior to the existing technology in terms of convergence.

[0071] The horizontal axis in Figure 2(c) represents the number of training rounds, a total of 16 rounds, and the vertical axis represents the Rouge-1 value. The curve marked with "+" represents the relationship between the Rouge-1 value and the number of training rounds for rewriting text using the semantically missing text rewriting network proposed in this invention, while the curve marked with "*" represents the relationship between the Rouge-1 value and the number of training rounds for rewriting text using the conventional T-Ptr-λ method.

[0072] As can be seen from Figure 2(c), with the increase of training rounds, the Rouge-1 value of the semantically missing text rewriting network proposed in the present invention is as high as 0.94, and the T-Ptr-λ of the prior art is as high as 0.89, which proves that the semantically missing text rewriting network proposed in the present invention is superior to the prior art in rewriting semantically missing text.

[0073] The horizontal axis in Figure 2(d) represents the number of training rounds, a total of 16 rounds, and the vertical axis represents the time consumed, in seconds. The curve marked with a "+" represents the relationship between the time consumed and the number of training rounds for rewriting the training set using the semantically missing text rewriting network proposed in this invention, and the curve marked with an "*" represents the relationship between the time consumed and the number of training rounds for rewriting the training set using the conventional T-Ptr-λ technique.

[0074] As can be seen from Figure 2(d), with the increase of training rounds, the rewriting time consumption of the semantically missing text rewriting network proposed in the present invention is stabilized at 15 seconds, and the rewriting time consumption of the prior art T-Ptr-λ is stabilized at 120 seconds, which proves that the semantically missing text rewriting network proposed in the present invention is superior to the prior art in terms of the rewriting time consumption of semantically missing text.

[0075] The above simulation experiments show that the present invention uses the Transformer RBT3 pre-trained model to process and extract text features, and uses a semantically missing text rewriting network constructed based on the idea of ​​extracting key information content from the text based on pointer addresses, which solves the problem of low quality and long time consumption of rewriting user text in the existing technology. In addition, the present invention constructs a text semantic correlation recognition network to identify whether the semantics of the user's two rounds of input text are related, determine whether the user's current input text needs to be combined with the previous round of text for semantic understanding, and then determine whether the user's current input text needs to be rewritten, thus solving the problem of the existing technology that cannot determine whether the user's current input text needs to be rewritten.

Claims

1. A method for rewriting multi-round human-computer dialogues based on Transformer pointer extraction, characterized in that: Based on the Transformer pre-trained model, feature extraction and encoding are performed, and a text semantic relevance recognition network and a semantically missing text rewriting network are constructed and trained respectively. The steps of this dialogue rewriting method include the following: Step 1: Build a text semantic relevance recognition network: A four-layer text semantic relevance recognition network was constructed, with the following structure connected in series: data processing layer, semantic encoding layer, classification processing layer, and output layer. The semantic encoding layer was implemented using the BERT pre-training model. The word embedding vector dimension was set to 768, and the maximum allowed length of the text was set to 10. The classification processing layer consisted of two identical first classification units and a second classification unit in series. Each classification unit consisted of a Dropout module and a fully connected module in series. The Dropout ratio was set to 0.2, the output dimension of the fully connected module in the first classification unit was set to 8, and the output dimension of the fully connected module in the second classification unit was set to 2. The output layer consisted of Softmax units, and the output values ​​were all set to 0 or 1. Step 2: Build a semantic missing text rewriting network: A four-layer semantically missing text rewriting network was constructed, with a structure consisting of a data processing layer, a semantic encoding layer, a pointer prediction layer, and an output layer. The semantic encoding layer was implemented using the Transformer RBT3 pre-trained model, with the word embedding vector dimension set to 768 and the maximum allowed text length set to 50. The pointer prediction layer was implemented using a pointer extraction method, which extracted the pointer addresses of key information in the text and replaced the referents in the rewritten text with the keywords extracted from the pointer addresses or inserted them into the default positions to obtain the rewritten complete semantic text. The training sample batch size was set to 16, and the total training round size was set to 20. The output layer was implemented by a decoder, and the network's rewriting evaluation metric for the sample data was set to the Rouge-1 value. Step 3: Generate training set: Step 3.1: Combine the two texts and their corresponding semantic relevance labels into a set of text semantic relevance recognition data in the format of (label, q1, q2), where q1 and q2 represent the two texts used for semantic relevance recognition, and label represents a label with a value of 0 or 1, corresponding to whether the two texts are semantically unrelated or related, respectively. Using the same method, select at least 80,000 sets of text semantic relevance recognition data to form a text semantic relevance recognition training set. Step 3.2: Group the four texts into a set of data in the format of (a, b, current, right), where a represents the user's input text in the previous round, b represents the system's reply text in the previous round, current represents the text to be rewritten, and right represents the theoretically correct rewritten text. Using the same method, select at least 17,000 sets of data to form the semantically missing text rewriting training set. Step 4: Train the text semantic relevance recognition network: The text semantic relevance recognition training set is input into the text semantic relevance recognition network, and the predicted labels of the training samples are output. The cross entropy loss function is used to calculate the loss value between the predicted labels and the true labels of the training samples. All parameters in the text semantic relevance recognition network are iteratively updated through the gradient descent method until the loss value no longer changes, thereby obtaining a trained text semantic relevance recognition network. Step 5: Train the semantic missing text rewriting network: Input the semantic missing text rewriting training set into the semantic missing text rewriting network, output the rewritten complete semantic text, and calculate the Rouge-1 value between the network-rewritten text and the theoretically correct rewritten text. Using the gradient descent method, iteratively update all parameters in the semantic missing text rewriting network until the Rouge-1 value no longer changes. Record all parameters of the training round with the highest Rouge-1 value to obtain the trained semantic missing text rewriting network. Step 6: Determine whether the user input text is semantically related: The user's previous input text and the user's current input text are input into the trained text semantic relevance recognition network, and the semantic relevance prediction label of the two rounds of user conversation text is output. If the predicted label value is 0, it means that the user's current input text has no semantic relevance with the previous round of input text and no rewriting is required. If the predicted label value is 1, it means that the user's current input text is semantically related to the previous round of input text and rewriting is required to understand the semantics of the current input text, and then execute step 7; Step 7: Rewrite the semantically missing text: The user's previous input text, the user's current input text, and the machine's previous reply text are all input into the trained semantically missing text rewriting network, and the complete semantic text after rewriting the user's current input text is output. The semantics of the user's current input text is understood based on the rewritten complete semantic text.

2. The method for rewriting multi-round human-computer dialogues based on Transformer pointer extraction according to claim 1 is characterized in that: The pointer addresses described in step 2 include a keyword position start pointer, a keyword position end pointer, a default position pointer, a pronoun position start pointer, and a pronoun position end pointer.

3. The method for rewriting multi-round human-computer dialogues based on Transformer pointer extraction according to claim 1 is characterized in that: The pointer extraction method described in step 2 is to obtain keywords, pronouns, and default positions by comparing the differences between the current text to be rewritten and the theoretically correctly rewritten text in the training set. The start pointer and end pointer of the keyword position in the user's previous round of input text are extracted based on the keywords, the start pointer and end pointer of the pronoun position in the current text to be rewritten are extracted based on the pronouns, and the default position pointer in the current text to be rewritten is extracted based on the default position.

4. The method for rewriting multi-round human-computer dialogues based on Transformer pointer extraction according to claim 1 is characterized in that: The loss between the predicted label and the true label of the training sample in step 4 is obtained by the following formula: Among them, Loss represents the loss value between the predicted label and the true label of the training sample, y i Represents the network's predicted label value for the i-th group of data, label represents the true label value of the i-th group of data, criterion represents the cross entropy function, n represents the total number of data in the i-th group of training samples, ∑ represents the summation operation, and size represents the size of the training sample batch.

5. The method for rewriting multi-round human-computer dialogues based on Transformer pointer extraction according to claim 4 is characterized in that: The Rouge-1 value described in step 5 is obtained by the following formula: Among them, rouge represents the Rouge-1 value of the semantic missing text rewriting network rewriting the i-th group of data, y i Represents the words in the text after the network rewrites the i-th group of data, represents the words in the text that are correctly rewritten by the i-th group of data, and count represents the statistics y i and The same number, len(gram) represents the length of the text that can be theoretically correctly rewritten for the i-th group of data.

6. The method for rewriting multi-round human-computer dialogues based on Transformer pointer extraction according to claim 1 is characterized in that: The machine's last reply text in step 7 refers to the text that the machine replied to after searching for the answer to the user's last input text. It belongs to the conversation history text between the user and the machine and is stored in the Redis third-party middleware.

Images