A paraphrase generation method, device, equipment and storage medium

Abstract

The application discloses a kind of paraphrase generation method, device, equipment and storage medium, method includes obtaining first paraphrase generation corpus and word segmentation processing, the input word sequence X_1 and label word sequence Y_1 obtained are used as pre-training data set to train neural network model M;Second paraphrase generation corpus is obtained and is constructed knowledge base by neural network model M, so that the paraphrase generation knowledge contained in the second paraphrase generation corpus comprising first paraphrase generation corpus and incremental paraphrase generation corpus with timeliness exists in the form of key-value pair in knowledge base, the input word sequence X_3 obtained by third paraphrase generation corpus word segmentation processing is input into neural network model M to predict, obtain neural network prediction result and query vector;Query vector is used to retrieve knowledge base, and obtain retrieval result;Fusion neural network prediction result and retrieval result, generate final paraphrase text.Knowledge base makes paraphrase system effective iteration update, and generates paraphrase text with decision basis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing and machine learning counting, in particular to a paraphrase generation method and device, equipment and storage medium. BACKGROUND

[0002] Paraphrase refers to expressing a piece of text (sentence) in another way without changing its semantic meaning by using different words, phrases or other language structures. In natural language, paraphrase plays an important role, which is not only an important module in NLP (Natural Language Processing) system, but also an indispensable part of daily life.

[0003] The existing paraphrase generation mainly trains a sequence-to-sequence neural network model. First, the text is preprocessed, and after tokenization and vectorization, the word vector sequence is fed into the neural network as input. Finally, the real word label and the prediction given by the model are compared, the gradient is calculated using the back propagation algorithm, and the neural network weight is updated to make the network fit the distribution of the training set data. However, this neural network training method is not suitable for scenarios where data continues to increase, and it also lacks certain explainability.

[0004] The above method specifically applies deep learning technology, uses an end-to-end generation model or a reinforcement learning model, and achieves good paraphrase generation results. However, the model based on deep learning technology has a difficult defect to solve: it cannot well face the scenario of continuous increase of training data. In actual application, the data used to build the paraphrase system is not obtained at the same time, nor is it unchangeable. With the passage of time, the collected data is a cumulative process. However, the neural network used by this method needs to be trained on a fixed training set, and cannot adapt to fit the later data. Even if the fine-tuning technology is used, it will encounter an unacceptable catastrophic forgetting problem, that is, it adapts to the later data but forgets the previous data. SUMMARY

[0005] To overcome the shortcomings of the existing neural network training method in the background technology, that is, the neural network used needs to be trained on a fixed training set, and cannot adapt to fit the later data, even if the fine-tuning technology is used, it will encounter an unacceptable catastrophic forgetting problem, that is, it adapts to the later data but forgets the previous data. This method is not suitable for scenarios where data continues to increase, and it also lacks certain explainability. The present application provides a paraphrase generation method.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A first aspect of the present invention provides a restatement generation method, comprising:

[0008] Obtain the first paraphrased corpus, perform word segmentation on the first paraphrased corpus to obtain the input word sequence X_1 and the label word sequence Y_1, and use the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset to train the neural network model M.

[0009] Obtain the second paraphrase generation corpus, construct a knowledge base using the second paraphrase generation corpus and neural network model M, so that the paraphrase generation knowledge contained in the second paraphrase generation corpus exists in the knowledge base in the form of key-value pairs, and the second paraphrase generation corpus includes the first paraphrase generation corpus and the time-sensitive incremental paraphrase generation corpus;

[0010] A third paraphrase generation corpus is obtained, and the third paraphrase generation corpus is segmented to obtain an input word sequence X_3. The input word sequence X_3 is input into the neural network model M for prediction to obtain the neural network prediction result and the query vector. The query vector is used to retrieve the knowledge base to obtain the retrieval result. The neural network prediction result and the retrieval result are fused to generate the final paraphrase text.

[0011] The beneficial effects of this invention are as follows:

[0012] 1. From a technical perspective, this invention adds a pluggable knowledge base to the predictive text generated by the neural network model M. It does not rely solely on the output of the neural network for prediction, but uses the knowledge base as an aid. Since the knowledge base can be easily added to and removed from, the entire paraphrasing system has strong scalability. At the same time, the results of the knowledge base retrieval can be used as the basis for the final paraphrased text generation, which increases the interpretability of the system.

[0013] 2. From an application perspective, this invention is more in line with real-world data collection scenarios, enabling the paraphrasing system to effectively iterate and update as incremental learning data is acquired. Through knowledge base retrieval, it assists the neural network model in generating the final paraphrased text that provides users with decision-making support.

[0014] In some possible implementations, the neural network model M employs a Transformer model consisting of an encoder and a decoder; training the neural network model M using the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset specifically includes the following steps:

[0015] S11: The input word sequence X_1 is input into the encoder. The input word sequence X_1 is multiplied by the word embedding matrix Encoder_embedding to obtain the word vector matrix Emb_x. The word vector matrix Emb_x is then multiplied by the three parameter matrices of the encoder Encoder_weight_q, Encoder_weight_k, and Encoder_weight_v to obtain three matrices: Encoder_query, Encoder_key, and Encoder_value. The encoder obtains the encoder self-attention result Encoder_attn using a self-attention algorithm, as follows:

[0016] Encoder_attn=softmax(Encoder_query×Encoder_key / √d)×

[0017] Encoder_value,

[0018] Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the encoder self-attention result Encoder_attn is a vector sequence carrying encoding end context information;

[0019] S12: The decoder uses the calculated encoder self-attention result Encoder_attn to sequentially calculate the prediction probability at each position to be generated. Specifically, when predicting a word, the word sequence Y_prev generated by word segmentation is used as the input to the decoder. The word sequence Y_prev is multiplied by the decoder's word embedding matrix Decoder_embedding to obtain the word vector matrix Emb_y_prev. The word vector matrix Emb_y_prev is then multiplied by the decoder's three parameter matrices Decoder_weight_q, Decoder_weight_k, and Decoder_weight_v to obtain three matrices: Decoder_query, Decoder_key, and Decoder_value. The decoder uses a self-attention algorithm to obtain the decoder's self-attention result Decoder_attn, as shown in the following formula:

[0020] Decoder_attn=softmax(Decoder_query×Decoder_key / √d)×

[0021] Decoder_value,

[0022] Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder self-attention result Decoder_attn is a vector sequence carrying decoding context information.

[0023] The cross-attention result Cross_attn is calculated from the decoder self-attention result Decoder_attn and the encoder self-attention result Encoder_attn using the cross-attention algorithm formula, as follows:

[0024] Cross_attn=softmax((Decoder_weight_q×Decoder_attn)×

[0025] (Encoder_weight_k×Encoder_attn) / √d)×(Encoder_weight_v×

[0026] Encoder_attn),

[0027] Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder cross-attention result Cross_attn is a vector sequence that carries context information from both the encoder and decoder ends.

[0028] The Cross_attn vector, the result of the cross attention, is processed by the projection layer of the decoder and the softmax function to obtain the probability distribution P_predict of the vocabulary size.

[0029] S13: Convert the label words corresponding to the label word sequence Y_1 into a one-hot probability distribution P_label. Calculate the cross-entropy loss of the neural network model M using the probability distribution P_label and the probability distribution P_predict. Use the backpropagation algorithm to calculate the gradient of the neural network model M from the cross-entropy loss. Update the network weights (Weight) of the neural network model M using the gradient. The formula for calculating Weight is as follows:

[0030] Weight=Weight-learning_rate*Gradient,

[0031] Wherein, learning_rate is the learning rate.

[0032] In some possible implementations, a knowledge base is constructed using the second paraphrasing corpus and the neural network model M, so that the paraphrasing knowledge contained in the second paraphrasing corpus exists in the knowledge base in the form of key-value pairs. Specifically, this includes the following steps:

[0033] S21: The second paraphrased corpus is segmented to obtain the input word sequence X_2 and the tag word sequence Y_2;

[0034] S22: Feed the input word sequence X_2 of the second paraphrased corpus to the neural network model M, extract the input vector V_build of the projection layer of the decoder, and use the input vector V_build as the key of the knowledge base;

[0035] S23: The tag word sequence Y_2 of the second paraphrase generated corpus is used as the value Value of the knowledge base. The value Value represents the correct predicted tag. The key and the value Value are matched one by one according to the position of the tag word to form a set of key-value pairs.

[0036] S24: Create an index for the key-value pairs to complete the construction of the knowledge base.

[0037] In some possible implementations, the input word sequence X_3 of the third paraphrasing corpus is input into the neural network model M for prediction to obtain a prediction result and a query vector; the knowledge base is retrieved using the query vector to obtain a retrieval result; and the prediction result and the retrieval result are fused to generate the final paraphrased text. Specifically, this includes the following steps:

[0038] S31: Feed the input word sequence X_3 forward to the neural network model M. After the input word sequence X_3 passes through the encoder and decoder of the neural network model M in accordance with steps S11 to S12, the probability distribution P_model of the neural network model M in the current decoding step and the input vector V_test of the projection layer of the decoder are obtained.

[0039] S32: Use the input vector V_test as a query vector to retrieve nearest neighbor items in the knowledge base, thereby retrieving key-value pairs that are close to the input vector V_test in Euclidean distance.<Key_test,Value_test> and the corresponding search distance Distance_test, through the key-value pair<Key_test,Value_test> The retrieval probability distribution P_retrieve is constructed using the retrieval distance Distance_test. The formula for constructing P_retrieve is not fixed, but it should generally follow the principle that the greater the retrieval distance, the smaller the contribution probability of the retrieval item. A feasible formula is as follows:

[0040] P_retrieve=softmax(-Distance_test),

[0041] Here, softmax is a function that normalizes a numerical vector into a probability distribution vector, and Distance_test is the search distance for each word in the vocabulary. If a word in the vocabulary does not appear in Value_test of the search results, the search distance of that word is set to positive infinity.

[0042] S33: The final probability distribution P_final is obtained by weighted combination of the probability distribution P_model output by the neural network model M and the probability distribution P_retrieve constructed by knowledge base retrieval. The formula for P_final is as follows:

[0043] P_final=λ*P_retrieve+(1-λ)*P_model,

[0044] Wherein, λ is a hyperparameter with a value range of [0.0, 1.0]. The larger λ is, the greater the contribution of the probability distribution P_retrieve constructed by the retrieval to the final probability distribution P_final.

[0045] S34: The final probability distribution P_final is used to determine and generate the paraphrased text. Specifically, the word with the highest probability of P_final in a certain decoding step is the word that should be generated in that decoding step. By connecting the words that should be generated in each decoding step, the final paraphrased text is obtained.

[0046] A second aspect of the present invention provides a retelling generation apparatus, comprising:

[0047] Model training module: Obtain the first paraphrase generation corpus, perform word segmentation on the first paraphrase generation corpus to obtain the input word sequence X_1 and the label word sequence Y_1, and use the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset to train the neural network model M;

[0048] Knowledge base construction module: Obtain the second paraphrase generation corpus, perform word segmentation on the second paraphrase generation corpus to obtain input word sequence X_2 and tag word sequence Y_2, and use the input word sequence X_2, tag word sequence Y_2 and neural network model M to construct a knowledge base, so that the paraphrase generation knowledge contained in the second paraphrase generation corpus exists in the knowledge base in the form of key-value pairs. The second paraphrase generation corpus includes the first paraphrase generation corpus and the time-sensitive incremental paraphrase generation corpus.

[0049] Paraphrasing generation module: acquires the third paraphrasing generation corpus, segments the third paraphrasing generation corpus to obtain the input word sequence X_3, inputs the input word sequence X_3 into the neural network model M for prediction, and obtains the neural network prediction result and query vector; uses the query vector to search the knowledge base and obtain the search result; and fuses the neural network prediction result and the search result to generate the final paraphrased text.

[0050] In some possible implementations, the neural network model M employs a Transformer model consisting of an encoder and a decoder; training the neural network model M using the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset specifically includes the following steps:

[0051] S11: The input word sequence X_1 is input into the encoder. The input word sequence X_1 is multiplied by the word embedding matrix Encoder_embedding to obtain the word vector matrix Emb_x. The word vector matrix Emb_x is then multiplied by the three parameter matrices of the encoder Encoder_weight_q, Encoder_weight_k, and Encoder_weight_v to obtain three matrices: Encoder_query, Encoder_key, and Encoder_value. The encoder obtains the encoder self-attention result Encoder_attn using a self-attention algorithm, as follows:

[0052] Encoder_attn=softmax(Encoder_query×Encoder_key / √d)×

[0053] Encoder_value,

[0054] Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the encoder self-attention result Encoder_attn is a vector sequence carrying encoding end context information;

[0055] S12: The decoder uses the calculated encoder self-attention result Encoder_attn to sequentially calculate the prediction probability at each position to be generated. Specifically, when predicting a word, the word sequence Y_prev generated by word segmentation is used as the input to the decoder. The word sequence Y_prev is multiplied by the decoder's word embedding matrix Decoder_embedding to obtain the word vector matrix Emb_y_prev. The word vector matrix Emb_y_prev is then multiplied by the decoder's three parameter matrices Decoder_weight_q, Decoder_weight_k, and Decoder_weight_v to obtain three matrices: Decoder_query, Decoder_key, and Decoder_value. The decoder uses a self-attention algorithm to obtain the decoder's self-attention result Decoder_attn, as shown in the following formula:

[0056] Decoder_attn=softmax(Decoder_query×Decoder_key / √d)×

[0057] Decoder_value,

[0058] Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder self-attention result Decoder_attn is a vector sequence carrying decoding context information.

[0059] The cross-attention result Cross_attn is calculated from the decoder self-attention result Decoder_attn and the encoder self-attention result Encoder_attn using the cross-attention algorithm formula, as follows:

[0060] Cross_attn=softmax((Decoder_weight_q×Decoder_attn)×

[0061] (Encoder_weight_k×Encoder_attn) / √d)×(Encoder_weight_v×

[0062] Encoder_attn),

[0063] Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder cross-attention result Cross_attn is a vector sequence that carries context information from both the encoder and decoder ends.

[0064] The Cross_attn vector, the result of the cross attention, is processed by the projection layer of the decoder and the softmax function to obtain the probability distribution P_predict of the vocabulary size.

[0065] S13: Convert the label words corresponding to the label word sequence Y_1 into a one-hot probability distribution P_label. Calculate the cross-entropy loss of the neural network model M using the probability distribution P_label and the probability distribution P_predict. Use the backpropagation algorithm to calculate the gradient of the neural network model M from the cross-entropy loss. Update the network weights (Weight) of the neural network model M using the gradient. The formula for calculating Weight is as follows:

[0066] Weight=Weight-learning_rate*Gradient,

[0067] Wherein, learning_rate is the learning rate.

[0068] In some possible implementations, a knowledge base is constructed using the input word sequence X_2, the tag word sequence Y_2, and the neural network model M, so that the paraphrase generation knowledge contained in the second paraphrase generation corpus exists in the knowledge base in the form of key-value pairs. Specifically, this includes the following steps:

[0069] S21: Feed the input word sequence X_2 of the second paraphrased corpus forward into the neural network model M, extract the input vector V_build of the projection layer of the decoder, and use the input vector V_build as the key of the knowledge base;

[0070] S22: The tag word sequence Y_2 of the second paraphrase generated corpus is used as the value Value of the knowledge base. The value Value represents the correct predicted tag. The key and the value Value are matched one by one according to the position of the tag word to form a set of key-value pairs.

[0071] S23: Create an index for the key-value pairs to complete the construction of the knowledge base.

[0072] In some possible implementations, the input word sequence X_3 of the third paraphrasing corpus is input into the neural network model M for prediction to obtain a prediction result and a query vector; the knowledge base is retrieved using the query vector to obtain a retrieval result; and the prediction result and the retrieval result are fused to generate the final paraphrased text. Specifically, this includes the following steps:

[0073] S31: Feed the input word sequence X_3 forward to the neural network model M. After the input word sequence X_3 passes through the encoder and decoder of the neural network model M in accordance with steps S11 to S12, the probability distribution P_model of the neural network model M in the current decoding step and the input vector V_test of the projection layer of the decoder are obtained.

[0074] S32: Use the input vector V_test as a query vector to retrieve nearest neighbor items in the knowledge base, thereby retrieving key-value pairs that are close to the input vector V_test in Euclidean distance.<Key_test,Value_test> and the corresponding search distance Distance_test, through the key-value pair<Key_test,Value_test> The retrieval probability distribution P_retrieve is constructed using the retrieval distance Distance_test. The formula for constructing P_retrieve is not fixed, but it should generally follow the principle that the greater the retrieval distance, the smaller the contribution probability of the retrieval item. A feasible formula is as follows:

[0075] P_retrieve=softmax(-Distance_test),

[0076] Here, softmax is a function that normalizes a numerical vector into a probability distribution vector, and Distance_test is the search distance for each word in the vocabulary. If a word in the vocabulary does not appear in Value_test of the search results, the search distance of that word is set to positive infinity.

[0077] S33: The final probability distribution P_final is obtained by weighted combination of the probability distribution P_model output by the neural network model M and the probability distribution P_retrieve constructed by knowledge base retrieval. The formula for P_final is as follows:

[0078] P_final=λ*P_retrieve+(1-λ)*P_model,

[0079] Wherein, λ is a hyperparameter with a value range of [0.0, 1.0]. The larger λ is, the greater the contribution of the probability distribution P_retrieve constructed by the retrieval to the final probability distribution P_final.

[0080] S34: The final probability distribution P_final is used to determine and generate the paraphrased text. Specifically, the word with the highest probability of P_final in a certain decoding step is the word that should be generated in that decoding step. By connecting the words that should be generated in each decoding step, the final paraphrased text is obtained.

[0081] A third aspect of the present invention provides a computer device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the paraphrasing generation method as described above.

[0082] In a fourth aspect, the present invention provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the paraphrase generation method as described above. Attached Figure Description

[0083] Figure 1 This is a flowchart illustrating the overall steps of the generation method as described in an embodiment of the present invention;

[0084] Figure 2 Flowchart of the steps for constructing a knowledge base and establishing an index for embodiments of the present invention;

[0085] Figure 3 This is a flowchart illustrating the final text generation steps in an embodiment of the present invention.

[0086] Figure 4 This is a schematic diagram of the structure of the generating apparatus according to an embodiment of the present invention;

[0087] Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0088] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0089] See appendix Figure 1 As shown, a paraphrase generation method in this embodiment includes:

[0090] S1: Obtain the first paraphrased corpus, perform word segmentation on the first paraphrased corpus to obtain the input word sequence X_1 and the label word sequence Y_1, and use the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset to train the neural network model M.

[0091] In some implementations, the neural network model M employs a Transformer model consisting of an encoder and a decoder.

[0092] Training the neural network model M using the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset specifically includes the following steps:

[0093] S11: The input word sequence X_1 is input to the encoder. The input word sequence X_1 is multiplied by the word embedding matrix Encoder_embedding to obtain the word vector matrix Emb_x. The word vector matrix Emb_x is then multiplied by the three parameter matrices of the encoder Encoder_weight_q, Encoder_weight_k, and Encoder_weight_v to obtain three matrices: Encoder_query, Encoder_key, and Encoder_value. The encoder calculates the encoder self-attention result Encoder_attn using a self-attention algorithm. The formula for the self-attention algorithm is as follows:

[0094] Encoder_attn=softmax(Encoder_query×Encoder_key / √d)×

[0095] Encoder_value,

[0096] Here, d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the encoder self-attention result Encoder_attn is a vector sequence carrying encoding context information.

[0097] S12: The decoder uses the calculated encoder self-attention result Encoder_attn to sequentially calculate the prediction probability at each position to be generated. Specifically, when predicting a word, the previously generated word sequence Y_prev is used as the input to the decoder. The generated word sequence Y_prev is multiplied by the decoder's word embedding matrix Decoder_embedding to obtain the word vector matrix Emb_y_prev. The word vector matrix Emb_y_prev is then multiplied by the decoder's three parameter matrices Decoder_weight_q, Decoder_weight_k, and Decoder_weight_v to obtain...

[0098] The decoder comprises three matrices: Decoder_query, Decoder_key, and Decoder_value. The decoder uses a self-attention algorithm to calculate the self-attention result, Decoder_attn. The formula for the self-attention algorithm is as follows:

[0099] Decoder_attn=softmax(Decoder_query×Decoder_key / √d)×

[0100] Decoder_value,

[0101] Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder self-attention result Decoder_attn is a vector sequence carrying decoding context information.

[0102] The cross-attention result Cross_attn is calculated on the decoder self-attention result Decoder_attn and the encoder self-attention result Encoder_attn using the cross-attention algorithm formula as follows:

[0103] Cross_attn=softmax((Decoder_weight_q×Decoder_attn)×

[0104] (Encoder_weight_k×Encoder_attn) / √d)×(Encoder_weight_v×

[0105] Encoder_attn),

[0106] Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder cross-attention result Cross_attn is a vector sequence that carries context information from both the encoder and decoder ends.

[0107] The Cross_attn vector, the result of the cross-attention algorithm, is processed by the projection layer and softmax function of the decoder to obtain the probability distribution P_predict of the vocabulary size.

[0108] S13: Convert the label words corresponding to the label word sequence Y_1 into a one-hot probability distribution P_label. Calculate the cross-entropy loss of the neural network model M using the probability distribution P_label and the probability distribution P_predict. Use the backpropagation algorithm to calculate the gradient of the neural network model M from the cross-entropy loss. Update the network weights (Weight) of the neural network model M using the gradient. The formula for calculating Weight is as follows:

[0109] Weight=Weight-learning_rate*Gradient,

[0110] Wherein, learning_rate is the learning rate.

[0111] The overall technical effect of S11 to S13 is to train a neural network model M with basic paraphrasing generation capability using the first paraphrasing generation corpus. The neural network model M has the ability to generate paraphrased text of average quality for a given input text.

[0112] S2: Obtain the second paraphrase generation corpus. Construct a knowledge base using the second paraphrase generation corpus and the neural network model M, so that the paraphrase generation knowledge contained in the second paraphrase generation corpus exists in the knowledge base in the form of key-value pairs. The number of key-value pairs in the knowledge base increases with the size of the second paraphrase generation corpus. The second paraphrase generation corpus not only includes the first paraphrase generation corpus, but also includes more timely incremental paraphrase generation corpus continuously collected from books and the Internet by the personnel maintaining the paraphrase system over time. See Appendix. Figure 2 As shown, S2 specifically includes the following steps:

[0113] S21: The second paraphrased corpus is segmented to obtain the input word sequence X_2 and the tag word sequence Y_2.

[0114] S22: Feed the input word sequence X_2 of the second paraphrased corpus to the neural network model M, extract the input vector V_build of the projection layer of the decoder, and use the input vector V_build as the key of the knowledge base;

[0115] S23: The tag word sequence Y_2 of the second paraphrase generated corpus is used as the value Value of the knowledge base. The value Value represents the correct predicted tag. The key and the value Value are matched one by one according to the position of the tag word to form a set of key-value pairs.

[0116] S24: Create an index for the key-value pairs to complete the construction of the knowledge base;

[0117] The overall technical effect of constructing the knowledge base using methods S21 to S24 is that the neural network model M is used to convert the paraphrasing knowledge contained in the second paraphrasing corpus into key-value pairs and store them in the knowledge base. This construction method makes it easy to add the knowledge contained in newly collected paraphrasing corpus to the knowledge base.

[0118] S3: Obtain the third paraphrase generation corpus (test set), perform word segmentation on the third paraphrase generation corpus to obtain the input word sequence X_3, input the input word sequence X_3 into the neural network model M for prediction, and obtain the neural network prediction result and query vector; use the query vector to search the knowledge base to obtain the search result; fuse the neural network prediction result and the search result to generate the final paraphrase text. See the attached flowchart for the flowchart of generating the final paraphrase text. Figure 3 As shown, S3 specifically includes the following steps:

[0119] S31: Feed the input word sequence X_3 forward to the neural network model M. After the input word sequence X_3 passes through the encoder and decoder of the neural network model M in accordance with steps S11 to S12, the probability distribution P_model of the neural network model M in the current decoding step and the input vector V_test of the projection layer of the decoder are obtained.

[0120] S32: Use the input vector V_test as a query vector to retrieve nearest neighbor items in the knowledge base, thereby retrieving key-value pairs that are close to the input vector V_test in Euclidean distance.<Key_test,Value_test> and the corresponding search distance Distance_test, through the key-value pair<Key_test,Value_test> The retrieval probability distribution P_retrieve is constructed using the retrieval distance Distance_test. The formula for constructing P_retrieve is not fixed, but it should generally follow the principle that the greater the retrieval distance, the smaller the contribution probability of the retrieval item. A feasible formula is as follows:

[0121] P_retrieve=softmax(-Distance_test)

[0122] Here, softmax is a function that normalizes a numerical vector into a probability distribution vector, and Distance_test is the search distance for each word in the vocabulary. If a word in the vocabulary does not appear in Value_test of the search results, the search distance of that word is set to positive infinity.

[0123] S33: The final probability distribution P_final is obtained by weighted combination of the probability distribution P_model output by the neural network model M and the probability distribution P_retrieve constructed by knowledge base retrieval. The formula for P_final is as follows:

[0124] P_final=λ*P_retrieve+(1-λ)*P_model,

[0125] Wherein, λ is a hyperparameter with a value range of [0.0, 1.0]. The larger λ is, the greater the contribution of the probability distribution P_retrieve constructed by the retrieval to the final probability distribution P_final.

[0126] S34: The final probability distribution P_final is used to determine and generate the paraphrased text. Specifically, the word with the highest probability of P_final in a certain decoding step is the word that should be generated in that decoding step. By connecting the words that should be generated in each decoding step, the final paraphrased text is obtained.

[0127] The overall technical effect of text generation methods S31-S34 is that it integrates the knowledge contained in the neural network model M and the knowledge retrieved from the knowledge base through a weighted combination of probability distributions. The entire paraphrasing system generates the final paraphrased text based on the knowledge from these two sources. Since the neural network model M is pre-trained on a large-scale paraphrasing corpus, the knowledge provided by the neural network model M provides the basic paraphrasing generation capability for the entire system during generation. The knowledge base stores the knowledge of newly collected paraphrasing corpus in the form of key-value pairs, which can provide assistance to the neural network model M during paraphrasing generation, enabling the entire paraphrasing system to generate high-quality paraphrased text even when faced with inputs with high timeliness.

[0128] The following are specific examples of building a knowledge base in S2 and generating paraphrased text in S3:

[0129] Suppose that when building the knowledge base, there are two sets of paraphrasing data A and B, each set of data consisting of an input word sequence X and a tag word sequence Y:

[0130] A_X: Hello, Xiaoming

[0131] A_Y: Hi, Xiaoming

[0132] B_X: Have you eaten?

[0133] B_Y: Have you eaten?

[0134] These two sets of data are fed into neural network model M respectively. The input vector V of the decoder projection layer of neural network model M is then used to form binary pairs (key-value pairs) with words in the label word sequences (i.e., A_Y and B_Y), and added to the knowledge base. The final knowledge base consists of the following 7 binary pairs:

[0135] <(1.3,2.6,…),“Hi”>

[0136] <(1.4,1.1,…),“,”>

[0137] <(5.4,3.6,…),“Xiaoming”>

[0138] <(2.3, 5.6, …), "You">

[0139] <(7.3, 1.6, …), "Eat">

[0140] <(2.3, 2.9, …), "already">

[0141] <(3.4, 4.5, …), "not">

[0142] For each binary pair, the first item is the input vector V of the decoder projection layer, and the second item is the label word at the corresponding position in the label word sequence.

[0143] After constructing the knowledge base, next, it demonstrates using the knowledge base to assist in generating paraphrased text. Assume the data to be tested is C, and the test data only has the input word sequence C_X, and the corresponding paraphrased text C_Y needs to be generated:

[0144] C_X: What are you doing

[0145] Send the text C_X into the neural network model M, and take out the input vector V of the decoder projection layer of the model M. Assume the value of the vector V at the first decoding step is (2.4, 5.7, …), then the binary pair retrieved from the knowledge base with the closest Euclidean distance will be <(2.3, 5.6, …), "You">, which will result in a relatively high probability corresponding to the word "You" in the retrieval probability P_retrieve constructed in this step.

[0146] As can be seen from S3, the retrieval probability distribution P_retrieve is an important part of the final probability distribution P_final. Therefore, the probability value of the word "You" in P_final is also likely to be relatively high. From this example, it can be seen that the paraphrasing knowledge stored in the knowledge base, when the neural network model M makes word predictions, the knowledge base retrieval assists the neural network model M to generate the final paraphrased text that can provide a decision-making basis for users.

[0147] This application also provides a paraphrase generation device that executes the above steps of paraphrase generation. See the appendix Figure 4 As shown, this device includes:

[0148] Model training module: Obtain the first paraphrase generation corpus, perform word segmentation on the first paraphrase generation corpus to obtain the input word sequence X_1 and the label word sequence Y_1, and use the input word sequence X_1 and the label word sequence Y_1 as the pre-training data set to train the neural network model M;

[0149] The knowledge base construction module acquires the second paraphrase generation corpus, segments the second paraphrase generation corpus to obtain the input word sequence X_2 and the tag word sequence Y_2, and uses the input word sequence X_2, the tag word sequence Y_2 and the neural network model M to construct a knowledge base, so that the paraphrase generation knowledge contained in the second paraphrase generation corpus exists in the knowledge base in the form of key-value pairs; the number of key-value pairs in the knowledge base increases with the size of the second paraphrase generation corpus, and the second paraphrase generation corpus not only includes the first paraphrase generation corpus, but also includes more timely paraphrase generation corpus continuously collected from books and the Internet by the personnel maintaining the paraphrase system over time;

[0150] Paraphrasing generation module: acquires the third paraphrasing generation corpus (test set), segments the third paraphrasing generation corpus into words to obtain the input word sequence X_3, inputs the input word sequence X_3 into the neural network model M for prediction, and obtains the neural network prediction result and query vector; uses the query vector to search the knowledge base to obtain the search result; and fuses the neural network prediction result and the search result to generate the final paraphrased text.

[0151] See appendix Figure 5 As shown, this application also provides a computer device (terminal or server), the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the paraphrase generation method as described above.

[0152] The computer device may be: a smartphone, tablet, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop, or desktop computer. The computer device may also be referred to by other names such as user equipment, portable computer device, laptop computer device, desktop computer device, etc.

[0153] Typically, computer equipment includes a processor and memory. The processor is called the CPU (Central Processing Unit).

[0154] The processor can also be:

[0155] A DSP (Digital Signal Processor) is a processor composed of large-scale or very large-scale integrated circuit chips used to perform specific signal processing tasks. It gradually developed to meet the needs of high-speed, real-time signal processing tasks. With the development of integrated circuit technology and digital signal processing algorithms, the implementation methods of digital signal processors are constantly changing, and their processing capabilities are continuously improving and expanding.

[0156] ASIC (Application Specific Integrated Circuit) refers to an integrated circuit designed and manufactured to meet the specific requirements of a user and the needs of a specific electronic system.

[0157] FPGA (Field Programmable Gate Array) is a further development based on programmable devices such as PAL (Programmable Array Logic) and GAL (Generic Array Logic). It emerged as a semi-custom circuit in the field of Application-Specific Integrated Circuits (ASICs), solving the shortcomings of custom circuits while overcoming the limitation of the limited gate count of original programmable devices.

[0158] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory are used to store at least one instruction, which is executed by a processor to implement the restatement generation method provided in the method embodiments of this application.

[0159] The memory further includes one or more programs stored in the memory, and the one or more programs include a paraphrase generation method provided in the embodiments of this application.

[0160] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by the processor to implement the paraphrase generation method provided in the above-described method embodiments.

[0161] The above embodiments are only for illustrating the technical concept and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it. They should not be used to limit the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A restatement generation method, characterized in that: include Obtain the first paraphrased corpus, perform word segmentation on the first paraphrased corpus to obtain the input word sequence X_1 and the label word sequence Y_1, and use the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset to train the neural network model M. Obtain the second paraphrase generation corpus, construct a knowledge base using the second paraphrase generation corpus and neural network model M, so that the paraphrase generation knowledge contained in the second paraphrase generation corpus exists in the knowledge base in the form of key-value pairs, and the second paraphrase generation corpus includes the first paraphrase generation corpus and the time-sensitive incremental paraphrase generation corpus; A third paraphrase generation corpus is obtained, and the third paraphrase generation corpus is segmented to obtain an input word sequence X_3. The input word sequence X_3 is input into the neural network model M for prediction to obtain the neural network prediction result and the query vector. The query vector is used to retrieve the knowledge base to obtain the retrieval result. The neural network prediction result and the retrieval result are fused to generate the final paraphrase text. The construction of a knowledge base using the second paraphrasing corpus and neural network model M, so that the paraphrasing knowledge contained in the second paraphrasing corpus exists in the knowledge base in the form of key-value pairs, specifically includes the following steps: S21: The second paraphrased corpus is segmented to obtain the input word sequence X_2 and the tag word sequence Y_2; S22: Feed the input word sequence X_2 of the second paraphrased corpus forward into the neural network model M, extract the input vector V_build of the projection layer of the decoder, and use the input vector V_build as the key of the knowledge base; S23: The tag word sequence Y_2 of the second paraphrase generated corpus is used as the value Value of the knowledge base. The value Value represents the correct predicted tag. The key and the value Value are matched one by one according to the position of the tag word to form a set of key-value pairs. S24: Create an index for the key-value pairs to complete the construction of the knowledge base.

2. The restatement generation method according to claim 1, characterized in that: The neural network model M employs a Transformer model consisting of an encoder and a decoder; training the neural network model M using the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset specifically includes the following steps: S11: The input word sequence X_1 is input into the encoder. The input word sequence X_1 is multiplied by the word embedding matrix Encoder_embedding to obtain the word vector matrix Emb_x. The word vector matrix Emb_x is then multiplied by the three parameter matrices of the encoder Encoder_weight_q, Encoder_weight_k, and Encoder_weight_v to obtain three matrices: Encoder_query, Encoder_key, and Encoder_value. The encoder obtains the encoder self-attention result Encoder_attn using a self-attention algorithm, as follows: Encoder_attn =softmax(Encoder_query×Encoder_key / √d)×Encoder_value, Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the encoder self-attention result Encoder_attn is a vector sequence carrying encoding end context information; S12: The decoder uses the calculated encoder self-attention result Encoder_attn to sequentially calculate the prediction probability at each position to be generated. Specifically, when predicting a word, the word sequence Y_prev generated by word segmentation is used as the input to the decoder. The word sequence Y_prev is multiplied by the decoder's word embedding matrix Decoder_embedding to obtain the word vector matrix Emb_y_prev. The word vector matrix Emb_y_prev is then multiplied by the decoder's three parameter matrices Decoder_weight_q, Decoder_weight_k, and Decoder_weight_v to obtain three matrices: Decoder_query, Decoder_key, and Decoder_value. The decoder uses a self-attention algorithm to obtain the decoder's self-attention result Decoder_attn, as shown in the following formula: Decoder_attn=softmax(Decoder_query×Decoder_key / √d)×Decoder_value, Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder self-attention result Decoder_attn is a vector sequence carrying decoding context information. The cross-attention result Cross_attn is calculated from the decoder self-attention result Decoder_attn and the encoder self-attention result Encoder_attn using the cross-attention algorithm formula, as follows: Cross_attn=softmax((Decoder_weight_q×Decoder_attn)×(Encoder_weight_k×Encoder_attn) / √d)×(Encoder_weight_v×Encoder_attn), Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder cross-attention result Cross_attn is a vector sequence that carries context information from both the encoder and decoder ends. The Cross_attn vector, the result of the cross attention, is processed by the projection layer of the decoder and the softmax function to obtain the probability distribution P_predict of the vocabulary size. S13: Convert the label words corresponding to the label word sequence Y_1 into a one-hot probability distribution P_label. Calculate the cross-entropy loss of the neural network model M using the probability distribution P_label and the probability distribution P_predict. Use the backpropagation algorithm to calculate the gradient of the neural network model M from the cross-entropy loss. Update the network weights (Weight) of the neural network model M using the gradient. The formula for calculating Weight is as follows: Weight= Weight-learning_rate * Gradient, Wherein, learning_rate is the learning rate.

3. The restatement generation method according to claim 1, characterized in that: The input word sequence X_3 of the third paraphrasing corpus is input into the neural network model M for prediction to obtain the prediction result and the query vector; the knowledge base is retrieved through the query vector to obtain the retrieval result; the prediction result and the retrieval result are fused to generate the final paraphrased text, specifically including the following steps: S31: Feed the input word sequence X_3 forward to the neural network model M. After the input word sequence X_3 passes through the encoder and decoder of the neural network model M in accordance with steps S11 to S12, the probability distribution P_model of the neural network model M in the current decoding step and the input vector V_test of the projection layer of the decoder are obtained. S32: Use the input vector V_test as the query vector to retrieve nearest neighbor items in the knowledge base, thereby retrieving key-value pairs that are close to the input vector V_test in Euclidean distance.<Key_test,Value_test> and the corresponding search distance Distance_test, through the key-value pair<Key_test,Value_test> The retrieval probability distribution P_retrieve is constructed using the retrieval distance Distance_test. The formula for constructing P_retrieve is not fixed, but it should generally follow the principle that the greater the retrieval distance, the smaller the contribution probability of the retrieval item. A feasible formula is as follows: P_retrieve=softmax(-Distance_test), Here, softmax is a function that normalizes a numerical vector into a probability distribution vector, and Distance_test is the search distance for each word in the vocabulary. If a word in the vocabulary does not appear in Value_test of the search results, the search distance of that word is set to positive infinity. S33: The final probability distribution P_final is obtained by weighted combination of the probability distribution P_model output by the neural network model M and the probability distribution P_retrieve constructed by knowledge base retrieval. The formula for P_final is as follows: P_final=λ*P_retrieve+(1-λ)*P_model, Wherein, λ is a hyperparameter with a value range of [0.0, 1.0]. The larger λ is, the greater the contribution of the probability distribution P_retrieve constructed by the retrieval to the final probability distribution P_final. S34: The final probability distribution P_final is used to determine and generate the paraphrased text. Specifically, the word with the highest probability of P_final in a certain decoding step is the word that should be generated in that decoding step. By connecting the words that should be generated in each decoding step, the final paraphrased text is obtained.

4. A retelling generation apparatus, characterized in that: include Model training module: Obtain the first paraphrase generation corpus, perform word segmentation on the first paraphrase generation corpus to obtain the input word sequence X_1 and the label word sequence Y_1, and use the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset to train the neural network model M; Knowledge base construction module: Obtain the second paraphrase generation corpus, perform word segmentation on the second paraphrase generation corpus to obtain input word sequence X_2 and tag word sequence Y_2, and use the input word sequence X_2, tag word sequence Y_2 and neural network model M to construct a knowledge base, so that the paraphrase generation knowledge contained in the second paraphrase generation corpus exists in the knowledge base in the form of key-value pairs. The second paraphrase generation corpus includes the first paraphrase generation corpus and the time-sensitive incremental paraphrase generation corpus. Paraphrasing generation module: acquires the third paraphrasing generation corpus, segments the third paraphrasing generation corpus into words to obtain the input word sequence X_3, inputs the input word sequence X_3 into the neural network model M for prediction, and obtains the neural network prediction result and query vector; uses the query vector to search the knowledge base to obtain the search result; and fuses the neural network prediction result and the search result to generate the final paraphrased text. The process of constructing a knowledge base using the input word sequence X_2, the label word sequence Y_2, and the neural network model M, so that the paraphrasing generation knowledge contained in the second paraphrasing generation corpus exists in the knowledge base in the form of key-value pairs, specifically includes the following steps: S21: Feed the input word sequence X_2 of the second paraphrased corpus forward into the neural network model M, extract the input vector V_build of the projection layer of the decoder, and use the input vector V_build as the key of the knowledge base; S22: The tag word sequence Y_2 of the second paraphrase generated corpus is used as the value Value of the knowledge base. The value Value represents the correct predicted tag. The key and the value Value are matched one by one according to the position of the tag word to form a set of key-value pairs. S23: Create an index for the key-value pairs to complete the construction of the knowledge base.

5. The restatement generation apparatus according to claim 4, characterized in that: The neural network model M employs a Transformer model consisting of an encoder and a decoder; training the neural network model M using the input word sequence X_1 and the label word sequence Y_1 as a pre-training dataset specifically includes the following steps: S11: The input word sequence X_1 is input into the encoder. The input word sequence X_1 is multiplied by the word embedding matrix Encoder_embedding to obtain the word vector matrix Emb_x. The word vector matrix Emb_x is then multiplied by the three parameter matrices of the encoder Encoder_weight_q, Encoder_weight_k, and Encoder_weight_v to obtain three matrices: Encoder_query, Encoder_key, and Encoder_value. The encoder obtains the encoder self-attention result Encoder_attn using a self-attention algorithm, as follows: Encoder_attn =softmax(Encoder_query×Encoder_key / √d)×Encoder_value, Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the encoder self-attention result Encoder_attn is a vector sequence carrying encoding end context information; S12: The decoder uses the calculated encoder self-attention result Encoder_attn to sequentially calculate the prediction probability at each position to be generated. Specifically, when predicting a word, the word sequence Y_prev generated by word segmentation is used as the input to the decoder. The word sequence Y_prev is multiplied by the decoder's word embedding matrix Decoder_embedding to obtain the word vector matrix Emb_y_prev. The word vector matrix Emb_y_prev is then multiplied by the decoder's three parameter matrices Decoder_weight_q, Decoder_weight_k, and Decoder_weight_v to obtain three matrices: Decoder_query, Decoder_key, and Decoder_value. The decoder uses a self-attention algorithm to obtain the decoder's self-attention result Decoder_attn, as shown in the following formula: Decoder_attn=softmax(Decoder_query×Decoder_key / √d)×Decoder_value, Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder self-attention result Decoder_attn is a vector sequence carrying decoding context information. The cross-attention result Cross_attn is calculated from the decoder self-attention result Decoder_attn and the encoder self-attention result Encoder_attn using the cross-attention algorithm formula, as follows: Cross_attn=softmax((Decoder_weight_q×Decoder_attn)×(Encoder_weight_k×Encoder_attn) / √d)×(Encoder_weight_v×Encoder_attn), Where d is a parameter equal to the width of the neural model M network, softmax is a function that normalizes the numerical vector into a probability distribution vector, and the decoder cross-attention result Cross_attn is a vector sequence that carries context information from both the encoder and decoder ends. The Cross_attn vector, the result of the cross attention, is processed by the projection layer of the decoder and the softmax function to obtain the probability distribution P_predict of the vocabulary size. S13: Convert the label words corresponding to the label word sequence Y_1 into a one-hot probability distribution P_label. Calculate the cross-entropy loss of the neural network model M using the probability distribution P_label and the probability distribution P_predict. Use the backpropagation algorithm to calculate the gradient of the neural network model M from the cross-entropy loss. Update the network weights (Weight) of the neural network model M using the gradient. The formula for calculating Weight is as follows: Weight= Weight-learning_rate * Gradient, Wherein, learning_rate is the learning rate.

6. The restatement generation apparatus according to claim 4, characterized in that: The input word sequence X_3 of the third paraphrasing corpus is input into the neural network model M for prediction to obtain the prediction result and the query vector; the knowledge base is retrieved through the query vector to obtain the retrieval result; the prediction result and the retrieval result are fused to generate the final paraphrased text, specifically including the following steps: S31: Feed the input word sequence X_3 forward to the neural network model M. After the input word sequence X_3 passes through the encoder and decoder of the neural network model M in accordance with steps S11 to S12, the probability distribution P_model of the neural network model M in the current decoding step and the input vector V_test of the projection layer of the decoder are obtained. S32: Use the input vector V_test as the query vector to retrieve nearest neighbor items in the knowledge base, thereby retrieving key-value pairs that are close to the input vector V_test in Euclidean distance.<Key_test,Value_test> and the corresponding search distance Distance_test, through the key-value pair<Key_test,Value_test> The retrieval probability distribution P_retrieve is constructed using the retrieval distance Distance_test. The formula for constructing P_retrieve is not fixed, but it should generally follow the principle that the greater the retrieval distance, the smaller the contribution probability of the retrieval item. A feasible formula is as follows: P_retrieve=softmax(-Distance_test), Here, softmax is a function that normalizes a numerical vector into a probability distribution vector, and Distance_test is the search distance for each word in the vocabulary. If a word in the vocabulary does not appear in Value_test of the search results, the search distance of that word is set to positive infinity. S33: The final probability distribution P_final is obtained by weighted combination of the probability distribution P_model output by the neural network model M and the probability distribution P_retrieve constructed by knowledge base retrieval. The formula for P_final is as follows: P_final=λ*P_retrieve+(1-λ)*P_model, Wherein, λ is a hyperparameter with a value range of [0.0, 1.0]. The larger λ is, the greater the contribution of the probability distribution P_retrieve constructed by the retrieval to the final probability distribution P_final. S34: The final probability distribution P_final is used to determine and generate the paraphrased text. Specifically, the word with the highest probability of P_final in a certain decoding step is the word that should be generated in that decoding step. By connecting the words that should be generated in each decoding step, the final paraphrased text is obtained.

7. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the paraphrase generation method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the paraphrase generation method as described in any one of claims 1 to 3.

Images