A Chinese question rewriting method based on a GPT network model

Abstract

The application belongs to the technical field of natural language processing, and particularly relates to a Chinese question rewriting method based on a GPT network model. In intelligent question answering, when the training set of the terms is small, the model generalization effect is poor, at this time, the similar questions of the Chinese questions, namely the terms, need to be manually added for expanding the term corpus, so as to improve the generalization of the model, and make the intelligent question answering task achieve better results. This paper proposes a Chinese question rewriting scheme, which is very suitable for generating text expansion of the data set in the intelligent question answering scene, so as to improve the question answering effect and reduce the cost of manually adding terms. The application realizes the Chinese question rewriting technical scheme based on the GPT model, and reduces the labor cost of manually adding terms in the intelligent question answering.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a Chinese question rewriting method based on the GPT network model. Background Technology

[0002] In the operation of intelligent customer service, especially in the execution of specific individual tasks, training data is generally scarce. When the training dataset is small, models trained with few samples are unlikely to achieve the expected results. Therefore, there is an urgent need for a Chinese question rewriting scheme that can effectively expand the limited dataset, enabling the model to achieve better results on more data. This aims to address the issues of expanding dialogue datasets and dialogue scripts in existing chatbots and intelligent customer service systems, thereby improving model performance. Based on this technical background, this application proposes a Chinese question rewriting technology scheme based on the GPT model. Summary of the Invention

[0003] This invention implements a Chinese question rewriting technique based on a GPT model trained on a large dataset. It fully leverages the simplicity of the GPT network model and the end-to-end nature of deep learning. The condition for using this invention is that the maximum length of the concatenated two questions input to the model from each training sample in the training set does not exceed 72. Compared to traditional Chinese question rewriting schemes, this invention fully utilizes the text generation characteristics of GPT, transforming the Chinese question rewriting scheme into an end-to-end text generation task. Apart from collecting parallel corpora of similar Chinese questions, training the model, and inference the model, no manual intervention is required in the intermediate stages.

[0004] A method for rewriting Chinese questions based on the GPT network model includes the following steps:

[0005] S1. Constructing the training dataset

[0006] The training dataset was constructed based on the similarity matching datasets publicly available in GitHub open-source projects. Text pairs of similar text were selected from the Chinese text dataset. Chinese question matching pairs were placed in a `train.txt` file, with each line containing two matching Chinese questions separated by single quotes. <sep>'Separate the data, perform deletion and filtering operations, set the maximum text length max_len to 72, that is, delete text with a total length greater than 72, and at the same time swap the order of the two data of text matching to amplify the two training samples. After sorting, the Chinese question matching training dataset is obtained.'

[0007] S2. Establish a vocabulary list

[0008] After removing duplicate characters from all characters in the Chinese question matching training data obtained in step S1, a word list sequence is established, denoted as Dict. The first key of Dict is the character index number, and the second value of Dict is the specific single character. Dict is {character index number 0:'[CLS]', 1:'[PAD]', 2:'[SEP]', ...}, where [CLS] is the text start character, [PAD] represents padding to the max_len length when the text length is less than the maximum length, [SEP] is the separator between the Chinese question and the question after rewriting the Chinese question, and max_len is the maximum length of each text data mentioned in step S1.

[0009] S3. Data and Model Adaptation

[0010] If a text sample in the training dataset obtained in step S1 is denoted as list A, and this text is less than the maximum length of 72, then list A is padded to the maximum length of 72 using [PAD], and then mapped to an index number list using the dictionary Dict, which becomes the input tensor X of the GPT model. The corresponding output Y is a tensor that is missing a start character and is converted into a tensor by the index number list, which serves as the optimization target Y of the model output. The same operation is performed on other texts in the training dataset to obtain the entire input data and optimization target, which are used as the input and optimization target of the subsequent model.

[0011] S4. Constructing the GPT network model structure

[0012] The model stipulates that the maximum text length is 72 and the embedding dimension is 256. For each input in step S3, the text is first embedded and represented by token_embedding and postional_embedding. At this time, the shape of the text embedding matrix is ​​[72, 256]. The text embedding is then input into the GPT network model, which is a multi-head attention mechanism with a mask and a feedforward fully connected neural network as a unit, repeated 6 times, that is, 6 identical network layers, to obtain the overall network model structure.

[0013] S5. Training Model

[0014] Based on the model structure set in step S4, and with the addition of a model optimizer and loss function, the model is trained. Specifically, the model input data obtained in steps S1, S2, and S3 is fed into the overall network model structure obtained in step S4. The model's ultimate goal is to minimize the loss function. First, forward propagation is performed based on the network structure, calculating the loss based on the predicted and actual outputs. Then, backpropagation is performed, continuously updating the network model parameters through the Adam optimizer. This process is repeated until the loss function value is reduced to a minimum and stabilizes. The parameter matrix at this point is then saved, yielding the optimal network model.

[0015] S6. Model Prediction

[0016] During the model prediction phase, the model parameters are randomly dropped out at a rate of 0.1. In the decoding phase, the probability of the character at each predicted position is sorted by size, and only the two highest probabilities are retained. Then, the highest of these two probabilities is selected. In the text generation phase, one of the two most probabilities is sampled, and its index is obtained from the dictionary. This index is the key value for the dic, and the corresponding character is then retrieved based on the key value. '[SEP]' is denoted as List_input, with a length of len_List. It is then padded to the maximum length of 72 using '[PAD]'. It is then mapped to an index tensor using the dictionary Dict mentioned in step S2 and used as the model input. The network model obtained in step S5 is used and fed into the network structure in step S4 for forward propagation to obtain the logits value. The softmax function is then used to calculate the character with the highest probability at the len_List position of the model output. The character is then randomly sampled and padded to the maximum length max_len again using '[PAD]'. It is then mapped to an index tensor using the dictionary Dict mentioned in step S2 and used as the model input to feed into the optimal network model obtained in step S5. At this point, the character with the highest probability at the len_List position of the model output is sampled. This process is repeated to output the corresponding rewritten Chinese question. Finally, all the predicted characters after '[SEP]' are concatenated together as the final output of the text prediction.

[0017] S7. Model Application

[0018] The current database contains questions (Qa) and answers (Answer) that are matching pairs of questions and their corresponding answers. By inputting Qa multiple times into the GPT Chinese question rewriting model, multiple representations of Qa can be generated.

[0019] This invention implements a Chinese question rewriting technique based on a GPT model trained on a large dataset, fully leveraging the simple structure of the GPT network model and the end-to-end nature of deep learning. The application conditions for this invention are: the maximum length of the concatenated two questions from each training sample input to the model should not exceed 72 characters; and a certain degree of error is acceptable when applied to projects. Compared to traditional Chinese question rewriting schemes, this invention fully utilizes the text generation characteristics inherent in GPT, transforming the Chinese question rewriting scheme into an end-to-end text generation task. Aside from collecting parallel corpora of similar Chinese questions, training the model, and developing the inference model, no manual intervention is required. Compared to traditional search-based speech enhancement methods, generative speech enhancement methods can generate speech not present in the training set, resulting in more comprehensive speech and better generalization. Attached Figure Description

[0020] Figure 1 is a schematic diagram of the GPT network model rewriting process. Detailed Implementation

[0021] A method for rewriting Chinese questions based on the GPT network model includes the following steps:

[0022] S1. Constructing the training dataset

[0023] The training dataset was constructed based on the similarity matching datasets publicly available in GitHub open-source projects. Text pairs of similar text were selected from the Chinese text dataset. Chinese question matching pairs were placed in a `train.txt` file, with each line containing two matching Chinese questions separated by single quotes. <sep>'Separate the data, perform deletion and filtering operations, set the maximum text length max_len to 72, that is, delete text with a total length greater than 72, and at the same time swap the order of the two data of text matching to amplify the two training samples. After sorting, the Chinese question matching training dataset is obtained.'

[0024] S2. Establish a vocabulary list

[0025] After removing duplicate characters from all characters in the Chinese question matching training data obtained in step S1, a word list sequence is established, denoted as Dict. The first key of Dict is the character index number, and the second value of Dict is the specific single character. Dict is {character index number 0:'[CLS]', 1:'[PAD]', 2:'[SEP]', ...}, where [CLS] is the text start character, [PAD] represents padding to the max_len length when the text length is less than the maximum length, [SEP] is the separator between the Chinese question and the question after rewriting the Chinese question, and max_len is the maximum length of each text data mentioned in step S1.

[0026] S3. Data and Model Adaptation

[0027] If a text sample in the training dataset obtained in step S1 is denoted as list A, and this text is less than the maximum length of 72, then list A is padded to the maximum length of 72 using [PAD], and then mapped to an index number list using the dictionary Dict, which becomes the input tensor X of the GPT model. The corresponding output Y is a tensor that is missing a start character and is converted into a tensor by the index number list, which serves as the optimization target Y of the model output. The same operation is performed on other texts in the training dataset to obtain the entire input data and optimization target, which are used as the input and optimization target of the subsequent model.

[0028] S4. Constructing the GPT network model structure

[0029] The model stipulates that the maximum text length is 72 and the embedding dimension is 256. For each input in step S3, the text is first embedded and represented by token_embedding and postional_embedding. At this time, the shape of the text embedding matrix is ​​[72, 256]. The text embedding is then input into the GPT network model, which is a multi-head attention mechanism with a mask and a feedforward fully connected neural network as a unit, repeated 6 times, that is, 6 identical network layers, to obtain the overall network model structure.

[0030] S5. Training Model

[0031] Based on the model structure set in step S4, and with the addition of a model optimizer and loss function, the model is trained. Specifically, the model input data obtained in steps S1, S2, and S3 is fed into the overall network model structure obtained in step S4. The model's ultimate goal is to minimize the loss function. First, forward propagation is performed based on the network structure, calculating the loss based on the predicted and actual outputs. Then, backpropagation is performed, continuously updating the network model parameters through the Adam optimizer. This process is repeated until the loss function value is reduced to a minimum and stabilizes. The parameter matrix at this point is then saved, yielding the optimal network model.

[0032] S6. Model Prediction

[0033] During the model prediction phase, the model parameters are randomly dropped out at a rate of 0.1. In the decoding phase, the probability of the character at each predicted position is sorted by size, and only the two highest probabilities are retained. Then, the highest of these two probabilities is selected. In the text generation phase, one of the two most probabilities is sampled, and its index is obtained from the dictionary. This index is the key value for the dic, and the corresponding character is then retrieved based on the key value. '[SEP]' is denoted as List_input, with a length of len_List. It is then padded to the maximum length of 72 using '[PAD]'. It is then mapped to an index tensor using the dictionary Dict mentioned in step S2 and used as the model input. The network model obtained in step S5 is used and fed into the network structure in step S4 for forward propagation to obtain the logits value. The softmax function is then used to calculate the character with the highest probability at the len_List position of the model output. The character is then randomly sampled and padded to the maximum length max_len again using '[PAD]'. It is then mapped to an index tensor using the dictionary Dict mentioned in step S2 and used as the model input to feed into the optimal network model obtained in step S5. At this point, the character with the highest probability at the len_List position of the model output is sampled. This process is repeated to output the corresponding rewritten Chinese question. Finally, all the predicted characters after '[SEP]' are concatenated together as the final output of the text prediction.

[0034] S7. Model Application

[0035] The current database contains questions (Qa) and answers (Answer) that are matching pairs of questions and their corresponding answers. By inputting Qa multiple times into the GPT Chinese question rewriting model, multiple representations of Qa can be generated.

[0036] The following is a detailed explanation using case studies:

[0037] A Chinese question rewriting method based on the GPT network model, the Chinese question rewriting method mainly includes:

[0038] S1. Construct a training dataset for similarity matching pairs: Refer to publicly available similarity matching datasets such as https: / / github.com / liucongg / NLPDataSet on GitHub to construct the training set. The construction method is as follows: Select text combinations of two similar Chinese texts and discard text combinations of dissimilar Chinese texts.

[0039] The Chinese question matching pairs are placed in a train.txt file, with each line containing two matching Chinese questions separated by a string. <sep>Separate the keys, for example, one of the training samples is "What brand of sun umbrella is good". <sep>The text "What brand of sun umbrella is good?" is used to perform a deletion and filtering operation. The maximum text length `max_len` is set to 72, meaning text with a total length greater than 72 is deleted. Simultaneously, the order of the two matching texts is swapped and expanded, for example, to "What brand of sun umbrella is good?". <sep>A training sample that asks "What brand of sun umbrella is good?" can be transformed into "What brand of sun umbrella is good?" <sep>"What are some good brands of sun umbrellas?" <sep>Two training samples of "What brand of sun umbrella is good", and after sorting, a total of 330,000 Chinese question matching training data sets are obtained.

[0040] S2. Build a vocabulary: After de-duplicating all the characters in the training data set of Chinese question matching obtained in step S1, build a vocabulary sequence, denoted as Dict. The previous key of Dict is the character index number, and the latter value of Dict is the specific single character, that is, Dict={0:‘[CLS]’,1:”[PAD]”,2:”[SEP]",3:”国”.......}, where [CLS] is the text start symbol, [PAD] represents that when the text length is not enough for the maximum length, [PAD] is used to fill to the max_len length, [SEP] is the separator between the Chinese question and the rewritten question of the Chinese question, and max_len is the maximum length of each text data mentioned in step S1.

[0041] S3. Adapt data and model: Suppose a certain text sample in the training data set obtained in step S1 is "What brand of sun umbrella is good <sep>If the text is "What brand of sun umbrella is good", then this text X is first transformed into ['[CLS]', 'What', 'brand', 'of', 'sun', 'umbrella', 'is', 'good', 'useful', '', '[SEP]', 'Sun', 'umbrella', 'What', 'brand', 'of', 'is', 'good'], denoted as list A. This text is less than the maximum length of 72. Then, list A is padded to the maximum length of 72 through [PAD]. Then, through the dictionary Dict, it is mapped into a list of index numbers, and then into the input GPT model Tensor tensor X. The corresponding Y output is the one missing the start symbol ['What', 'brand', 'of', 'sun', 'umbrella', 'is', 'good', 'useful', '[SEP]', 'Sun', 'umbrella', 'What', 'brand', 'of', 'is', 'good']. Through the list of index numbers, it becomes a Tensor tensor, which is used as the optimization target Y of the model output. The same operation is performed on other texts in the training dataset to obtain the entire training dataset, which is used as the model input and optimization target.

[0042] S4. Construct the GPT network model structure: The specific operations are as follows: The model assumes a text length of 72 and an embedding dimension of 256. For each input to the model in step S3, first, the text is embedded through token_embedding + postional_embedding. At this time, the matrix shape of the text embedding representation is [72, 256]. This text embedding representation is input into the GPT network model, that is, the multi-head attention mechanism with masking mask Multi_Head_Attention + the feed-forward fully connected neural network Feed_forward is used as a unit and repeated 6 times, that is, 6 layers of the same network, to obtain the overall network model structure.

[0043] S5. Train the model: According to the model structure set in step S4, and add a model optimizer and a loss function to train the model; The specific training process is as follows: Feed the model inputs obtained in steps S1, S2, and S3 into the overall network model structure obtained in step S4. The ultimate goal of the model is to minimize the loss function. First, perform forward propagation according to the network structure, calculate the loss based on the predicted output and the true output, and then perform backward propagation. Continuously update the network model parameters through the Adam optimizer, and repeat this process until the value of the loss function decreases to the minimum and stabilizes. Save the parameter matrix at this time to obtain the optimal network model.

[0044] S6. Model Prediction: In the model prediction stage, the random inactivation ratio dropout of the model parameters is 0.1, which remains the same as in the training stage. Dropout can make the rewritten Chinese questions of the model diverse. To further increase diversity, the decoding strategy is to sort the probabilities of the characters at each position predicted by the model in descending order, and only retain the first two largest tokens. Then, the largest one is retained from these two tokens. In the text generation stage, that is, one of the two tokens with the highest probability is sampled, and then the index is obtained through the dictionary. This index is the key value of the dictionary Dict, and then the corresponding character is obtained according to the key value. The example is as follows: The user's input question is "What brand of sun umbrella is good?". It is transformed into a sequence ['[CLS]', '什','么', '牌', '子', '的', '遮', '阳', '伞', '好', '用', '[SEP]'], denoted as List_input, and its length is len_List. Then it is padded to the maximum length of 72 with '[PAD]', and then mapped into an index number tensor through the dictionary Dict mentioned in step S2. This is used as the model input. Using the network model obtained in step S5, it enters the network structure in step S4 for forward propagation to obtain the logits value. Then, the softmax function is used to calculate the character with the highest probability at the len_List-th position of the model output, and then a random sample is taken. That is, at this time, the model predicts that the next character is "遮". Then, "遮" and the input are concatenated as ['[CLS]', '什','么', '牌', '子', '的', '遮', '阳', '伞', '好', '用', '[SEP]', '遮']. Then it is padded to the maximum length max_len again with '[PAD]', and then mapped into an index number tensor through the dictionary Dict mentioned in step S2. This is used as the model input and fed into the optimal network model obtained in step S5. At this time, a sample of the character with the highest probability at the (len_List + 1)-th position of the model output is calculated, which is "阳". Repeat this operation to output the corresponding rewritten Chinese questions in sequence. Finally, all the characters after [SEP] predicted are concatenated together as the final output result of the text prediction, that is, "What brand of sun umbrella is good?".

[0045] S7. Model Application: In intelligent question-answering systems, for example, if there are many question-answer pairs, and the database contains questions (Qa) and answers (Answer), by inputting Qa multiple times into the GPT Chinese question model, multiple representations of Qa can be generated, all with the same meaning. If the question-answering system is developed using a text classification model—that is, classifying user input into the corresponding answer category—a classification model needs to be trained. In this case, multiple ways of asking the same question are equivalent to data augmentation of the training dataset, which can improve the model's generalization ability.

[0046] The technical solution of this invention: This invention uses the GPT network model to rewrite Chinese questions. This solution only requires concatenating the collected Chinese question matching corpus pairs through the delimiter [SEP] and inputting them into the model for training. The GPT network model is only the decoder part of the Transformer, so it is simpler to implement than the Transformer.

[0047] The training parameters for the GPT model are as follows:

[0048]

[0049] The practicality of this application is as follows: by implementing a Chinese question rewriting scheme based on GPT, it can be well applied to intelligent question answering script enhancement, improving model performance and reducing labor costs.< / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep> < / sep>

Claims

1. A method for rewriting Chinese questions based on the GPT network model, characterized in that... Includes the following steps: S1. Constructing the training dataset The training dataset was constructed based on the similarity matching datasets publicly available in GitHub open-source projects. Text pairs of similar text were selected from the Chinese text dataset. Chinese question matching pairs were placed in a `train.txt` file, with each line containing two matching Chinese questions separated by single quotes. <sep> 'Separate the data, perform deletion and filtering operations, set the maximum text length max_len to 72, that is, delete text with a total length greater than 72, and at the same time swap the order of the two data of text matching to amplify the two training samples. After sorting, the Chinese question matching training dataset is obtained.'< / sep> S2. Establish a vocabulary list After removing duplicate characters from all characters in the Chinese question matching training data obtained in step S1, a word list sequence is established, denoted as Dict. The first key of Dict is the character index number, and the second value of Dict is the specific single character. Dict is {character index number 0:'[CLS]', 1:'[PAD]', 2:'[SEP]', ...}, where [CLS] is the text start character, [PAD] represents padding to the max_len length when the text length is less than the maximum length, [SEP] is the separator between the Chinese question and the question after rewriting the Chinese question, and max_len is the maximum length of each text data mentioned in step S1. S3. Data and Model Adaptation If a text sample in the training dataset obtained in step S1 is denoted as list A, and this text is less than the maximum length of 72, then list A is padded to the maximum length of 72 using [PAD], and then mapped to an index number list using the dictionary Dict, which becomes the input tensor X of the GPT model. The corresponding output Y is a tensor that is missing a start character and is converted into a tensor by the index number list, which serves as the optimization target Y of the model output. The same operation is performed on other texts in the training dataset to obtain the entire input data and optimization target, which are used as the input and optimization target of the subsequent model. S4. Constructing the GPT network model structure The model stipulates that the maximum text length is 72 and the embedding dimension is 256. For each input in step S3, the text is first embedded and represented by token_embedding and postional_embedding. At this time, the shape of the text embedding matrix is ​​[72, 256]. The text embedding is then input into the GPT network model, which is a multi-head attention mechanism with a mask and a feedforward fully connected neural network as a unit, repeated 6 times, that is, 6 identical network layers, to obtain the overall network model structure. S5. Training Model Based on the model structure set in step S4, and with the addition of a model optimizer and loss function, the model is trained. Specifically, the model input data obtained in steps S1, S2, and S3 is fed into the overall network model structure obtained in step S4. The model's ultimate goal is to minimize the loss function. First, forward propagation is performed based on the network structure, calculating the loss based on the predicted and actual outputs. Then, backpropagation is performed, continuously updating the network model parameters through the Adam optimizer. This process is repeated until the loss function value is reduced to a minimum and stabilizes. The parameter matrix at this point is then saved, yielding the optimal network model. S6. Model Prediction During the model prediction phase, the model parameters are randomly dropped out at a rate of 0.

1. In the decoding phase, the probability of the character at each predicted position is sorted by size, and only the two highest probabilities are retained. Then, the highest of these two probabilities is selected. In the text generation phase, one of the two most probabilities is sampled, and its index is obtained from the dictionary. This index is the key value for the dic, and the corresponding character is then retrieved based on the key value. '[SEP]' is denoted as List_input, with a length of len_List. It is then padded to the maximum length of 72 using '[PAD]'. It is then mapped to an index tensor using the dictionary Dict mentioned in step S2 and used as the model input. The network model obtained in step S5 is used and fed into the network structure in step S4 for forward propagation to obtain the logits value. The softmax function is then used to calculate the character with the highest probability at the len_List position of the model output. The character is then randomly sampled and padded to the maximum length max_len again using '[PAD]'. It is then mapped to an index tensor using the dictionary Dict mentioned in step S2 and used as the model input to feed into the optimal network model obtained in step S5. At this point, the character with the highest probability at the len_List position of the model output is sampled. This process is repeated to output the corresponding rewritten Chinese question. Finally, all the predicted characters after '[SEP]' are concatenated together as the final output of the text prediction. S7. Model Application The current database contains questions (Qa) and answers (Answer) that are matching pairs of questions and their corresponding answers. By inputting Qa multiple times into the GPT Chinese question rewriting model, multiple representations of Qa can be generated.

Images