Text reasoning method and device

Abstract

The present application provides a text reasoning method and device, wherein the text reasoning method includes: obtaining a text to be reasoned, inputting the text to be reasoned into an encoder, obtaining an encoding vector of the text to be reasoned, inputting the encoding vector and a target phrase into a first decoder, obtaining a first embedding vector, inputting the target phrase into a second decoder, obtaining a second embedding vector, summing the first embedding vector and the second embedding vector, obtaining an output word of current reasoning through an output layer from the sum result, updating the target phrase based on the output word of current reasoning, returning to execute inputting the encoding vector and the target phrase into the first decoder, obtaining a first embedding vector, inputting the target phrase into the second decoder, obtaining a second embedding vector, and, when the output word of current reasoning carries an end tag, determining that the output word in the target phrase constitutes the reasoning result of the text to be reasoned, thereby improving the accuracy of the output word of current reasoning, and thereby improving the accuracy of the text reasoning result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence in computer technology, and in particular to a text reasoning method. The present application also relates to a text reasoning device, a computing device, and a computer-readable storage medium. Background Art

[0002] Artificial intelligence (AI) refers to the ability of engineered (i.e. designed and manufactured) systems to perceive the environment, as well as the ability to acquire, process, apply and represent knowledge. Natural language processing, robotics and computer vision have become the three most popular industrial directions of artificial intelligence. The development status of key technologies in the field of artificial intelligence, including machine learning, knowledge graphs, natural language processing, computer vision, human-computer interaction, biometrics, virtual reality / augmented reality and other key technologies. Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. It involves many fields, mainly including machine translation, machine reading comprehension and question-answering systems. Machine translation technology refers to the use of computer technology to achieve the translation process from one natural language to another.

[0003] With the development of Internet technology, text reasoning is increasingly dependent on the Internet. Text reasoning is the process of analyzing, understanding, and extracting text. Text reasoning can help people perform text understanding, cluster analysis, abstract generation, information extraction, etc. Therefore, text reasoning has been widely used in various fields of people's daily life.

[0004] In the prior art, language models are generally used for text reasoning. Language models are generally pre-trained on large-scale corpora. When used, different components are connected according to downstream tasks to implement the text reasoning process. However, the current language model structure is single, and the accuracy of text reasoning results obtained based on language models is poor, which greatly affects the accuracy of text reasoning results. Summary of the invention

[0005] In view of this, the embodiment of the present application provides a text reasoning method to solve the technical defects existing in the prior art. The embodiment of the present application also provides a text reasoning device, a computing device, and a computer-readable storage medium.

[0006] According to a first aspect of an embodiment of the present application, a text reasoning method is provided, comprising:

[0007] Obtain the text to be inferred, input the text to be inferred into the encoder, and obtain the encoding vector of the text to be inferred;

[0008] Input the encoding vector and the target phrase into a first decoder to obtain a first embedding vector, and input the target phrase into a second decoder to obtain a second embedding vector, wherein the target phrase includes the inferred output word;

[0009] The first embedding vector and the second embedding vector are added together, and the sum is passed through the output layer to obtain the output word of the current reasoning;

[0010] Based on the output word of the current reasoning, update the target phrase, return to execute, input the encoding vector and the target phrase into the first decoder, obtain the first embedding vector, input the target phrase into the second decoder, and obtain the second embedding vector;

[0011] In the case where the output word of the current reasoning carries an end tag, it is determined that the output words in the target phrase constitute the reasoning result of the text to be reasoned.

[0012] Optionally, the second decoder is a pre-trained language model.

[0013] Optionally, the first character in the target phrase is a start tag;

[0014] In the initial case, the steps of inputting the encoding vector and the target phrase into the first decoder to obtain the first embedding vector, and inputting the target phrase into the second decoder to obtain the second embedding vector include:

[0015] Input the encoding vector and the start tag into the first decoder to obtain a first embedding vector, and input the start tag into the second decoder to obtain a second embedding vector;

[0016] The steps of updating the target phrase based on the output word of the current reasoning include:

[0017] The output word of the current reasoning is used as a new character and added to the target phrase;

[0018] The step of determining the output word composition in the target phrase as the reasoning result of the text to be reasoned includes:

[0019] The output words in the target phrase are merged in the order in which they are inferred to obtain the inference result of the text to be inferred.

[0020] Optionally, the step of adding the first embedding vector and the second embedding vector, and passing the sum result through the output layer to obtain the output word of the current reasoning includes:

[0021] Acquire a first weight coefficient preset for the first decoder and a second weight coefficient preset for the second decoder;

[0022] The first embedding vector and the second embedding vector are weighted by using the first weight coefficient and the second weight coefficient, and the weighted result is passed through the output layer to obtain the output word of the current reasoning.

[0023] Optionally, the training method of the encoder, the first decoder and the second decoder includes:

[0024] Obtaining a sample set, the sample set including a plurality of sample texts and target texts corresponding to the sample texts;

[0025] Extracting a first sample text, inputting the first sample text into an encoder, and obtaining a first sample encoding vector of the first sample text;

[0026] Inputting the first sample encoding vector and the first target phrase into a first decoder to obtain a first sample embedding vector, and inputting the first target phrase into a second decoder to obtain a second sample embedding vector, wherein the first target phrase includes an output word that has been inferred for the first sample text;

[0027] Add the first sample embedding vector and the second sample embedding vector, and pass the sum result through the output layer to obtain the first current output word;

[0028] Based on the first current output word, update the first target phrase, return to execute inputting the first sample encoding vector and the first target phrase into the first decoder to obtain the first sample embedding vector, inputting the first target phrase into the second decoder to obtain the second sample embedding vector;

[0029] In the case where the first current output word carries an end tag, determining that the output words in the first target phrase constitute a first inference result of the first sample text;

[0030] According to the first inference result and the first target text corresponding to the first sample text, the encoder, the first decoder and the second decoder are trained to obtain the trained encoder, the first decoder and the second decoder.

[0031] Optionally, the step of training the encoder, the first decoder, and the second decoder according to the first inference result and the first target text corresponding to the first sample text to obtain the trained encoder, the first decoder, and the second decoder includes:

[0032] Calculate a first difference value between the first inference result and the first target text corresponding to the first sample text;

[0033] If the first difference value is greater than a first preset threshold, the model parameters of the encoder, the first decoder and the second decoder are adjusted, and the first sample text is extracted again, and the first sample text is input into the encoder to obtain a first sample encoding vector of the first sample text;

[0034] If the first difference value is less than or equal to the first preset threshold, the training is stopped to obtain the trained encoder, the first decoder and the second decoder.

[0035] Optionally, the second decoder is pre-trained;

[0036] The training method of the encoder and the first decoder includes:

[0037] Obtaining a sample set, the sample set including a plurality of sample texts and target texts corresponding to the sample texts;

[0038] Extracting a second sample text, inputting the second sample text into an encoder, and obtaining a second sample encoding vector of the second sample text;

[0039] Inputting the second sample encoding vector and the second target phrase into the first decoder to obtain a third sample embedding vector, and inputting the second target phrase into the second decoder to obtain a fourth sample embedding vector, wherein the second target phrase includes an output word that has been inferred for the second sample text;

[0040] Add the third sample embedding vector and the fourth sample embedding vector, and pass the sum result through the output layer to obtain the second current output word;

[0041] Based on the second current output word, update the second target word group, return to execute inputting the second sample encoding vector and the second target word group into the first decoder to obtain a third sample embedding vector, inputting the second target word group into the second decoder to obtain a fourth sample embedding vector;

[0042] In the case where the second current output word carries an end tag, determining that the output words in the second target phrase constitute a second inference result of the second sample text;

[0043] The encoder and the first decoder are trained according to the second inference result and the second target text corresponding to the second sample text to obtain the trained encoder and the first decoder.

[0044] Optionally, the step of training the encoder and the first decoder according to the second inference result and the second target text corresponding to the second sample text to obtain the trained encoder and the first decoder includes:

[0045] Calculate a second difference value between the second inference result and the second target text corresponding to the second sample text;

[0046] If the second difference value is greater than the second preset threshold, the model parameters of the encoder and the first decoder are adjusted, the model parameters of the second decoder are kept unchanged, and the second sample text is extracted and input into the encoder to obtain a second sample encoding vector of the second sample text;

[0047] If the second difference value is less than or equal to the second preset threshold, the training is stopped to obtain the trained encoder and the first decoder.

[0048] According to a second aspect of an embodiment of the present application, a text reasoning device is provided, including:

[0049] The encoding vector obtaining module is configured to obtain the text to be inferred, input the text to be inferred into the encoder, and obtain the encoding vector of the text to be inferred;

[0050] An embedding vector obtaining module is configured to input the encoding vector and the target phrase into a first decoder to obtain a first embedding vector, and input the target phrase into a second decoder to obtain a second embedding vector, wherein the target phrase includes the output word that has been inferred;

[0051] An output word obtaining module is configured to add the first embedding vector and the second embedding vector, and obtain an output word of the current reasoning through the output layer by passing the sum result;

[0052] An update module is configured to update a target phrase based on an output word of the current reasoning, return to execute inputting the encoding vector and the target phrase into a first decoder to obtain a first embedding vector, and inputting the target phrase into a second decoder to obtain a second embedding vector;

[0053] The inference result determination module is configured to determine the inference result of the output word in the target phrase constituting the text to be inferred when the output word of the current inference carries an end tag.

[0054] Optionally, the second decoder is a pre-trained language model.

[0055] Optionally, the first character in the target phrase is a start tag;

[0056] In an initial situation, the embedding vector obtaining module is further configured to input the encoding vector and the start tag into a first decoder to obtain a first embedding vector, and input the start tag into a second decoder to obtain a second embedding vector;

[0057] The updating module is further configured to add the output word of the current reasoning as a new character to the target phrase;

[0058] The inference result determination module is further configured to merge the output words in the target phrase according to the order in which they are inferred, so as to obtain the inference result of the text to be inferred.

[0059] Optionally, the output word acquisition module is further configured to obtain a first weight coefficient pre-set for the first decoder and a second weight coefficient pre-set for the second decoder; use the first weight coefficient and the second weight coefficient to weight the first embedding vector and the second embedding vector, and obtain the output word of the current inference through the output layer by passing the weighted result.

[0060] Optionally, the device also includes: a first training module, configured to obtain a sample set, the sample set includes multiple sample texts and target texts corresponding to each sample text; extract the first sample text, input the first sample text into the encoder, and obtain a first sample encoding vector of the first sample text; input the first sample encoding vector and the first target phrase into the first decoder to obtain a first sample embedding vector, input the first target phrase into the second decoder to obtain a second sample embedding vector, wherein the first target phrase includes an output word that has been inferred for the first sample text; sum the first sample embedding vector and the second sample embedding vector, and obtain the first current output word through the output layer; based on the first current output word, update the first target phrase, return to execute inputting the first sample encoding vector and the first target phrase into the first decoder to obtain the first sample embedding vector, inputting the first target phrase into the second decoder to obtain the second sample embedding vector; when the first current output word carries an end tag, determine that the output words in the first target phrase constitute the first inference result of the first sample text; according to the first inference result and the first target text corresponding to the first sample text, train the encoder, the first decoder and the second decoder to obtain the trained encoder, the first decoder and the second decoder.

[0061] Optionally, the first training module is further configured to calculate a first difference value between the first inference result and the first target text corresponding to the first sample text; if the first difference value is greater than a first preset threshold, adjust the model parameters of the encoder, the first decoder and the second decoder, and return to execute the extraction of the first sample text, input the first sample text into the encoder, and obtain a first sample encoding vector of the first sample text; if the first difference value is less than or equal to the first preset threshold, stop training and obtain the trained encoder, first decoder and second decoder.

[0062] Optionally, the second decoder is pre-trained;

[0063] The device also includes: a second training module, which is configured to obtain a sample set, which includes multiple sample texts and target texts corresponding to each sample text; extract the second sample text, input the second sample text into the encoder, and obtain a second sample encoding vector of the second sample text; input the second sample encoding vector and the second target phrase into the first decoder to obtain a third sample embedding vector, input the second target phrase into the second decoder, and obtain a fourth sample embedding vector, wherein the second target phrase includes an output word that has been inferred for the second sample text; sum the third sample embedding vector and the fourth sample embedding vector, and obtain a second current output word through the output layer; based on the second current output word, update the second target phrase, return to execute inputting the second sample encoding vector and the second target phrase into the first decoder to obtain the third sample embedding vector, inputting the second target phrase into the second decoder to obtain the fourth sample embedding vector; in the case where the second current output word carries an end tag, determine that the output words in the second target phrase constitute a second inference result of the second sample text; according to the second inference result and the second target text corresponding to the second sample text, train the encoder and the first decoder to obtain the trained encoder and the first decoder.

[0064] Optionally, the second training module is further configured to calculate a second difference value between the second inference result and the second target text corresponding to the second sample text; if the second difference value is greater than a second preset threshold, the model parameters of the encoder and the first decoder are adjusted, the model parameters of the second decoder are kept unchanged, and the second sample text is extracted and input into the encoder to obtain a second sample encoding vector of the second sample text; if the second difference value is less than or equal to the second preset threshold, the training is stopped to obtain the trained encoder and first decoder.

[0065] According to a third aspect of an embodiment of the present application, a computing device is provided, including:

[0066] Memory and processor;

[0067] The memory is used to store computer-executable instructions, and the processor implements the steps of the text reasoning method when executing the computer-executable instructions.

[0068] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the text reasoning method are implemented.

[0069] According to a fifth aspect of an embodiment of the present application, a chip is provided, which stores computer instructions, and when the instructions are executed by the chip, the steps of the text reasoning method are implemented.

[0070] The text reasoning method provided by the present application can input the text to be reasoned into the encoder to obtain the encoding vector of the text to be reasoned, input the encoding vector and the target phrase into the first decoder to obtain the first embedding vector, input the target phrase into the second decoder to obtain the second embedding vector, sum the first embedding vector and the second embedding vector, and obtain the output word of the current reasoning through the output layer. Based on the output word of the current reasoning, update the target phrase, return to execute the encoding vector and the target phrase into the first decoder to obtain the first embedding vector, input the target phrase into the second decoder to obtain the second embedding vector, and determine the output word in the target phrase to constitute the reasoning result of the text to be reasoned when the output word of the current reasoning carries an end tag. In this case, by inputting the target phrase into the second decoder to obtain the second embedding vector, summing the first embedding vector and the second embedding vector to obtain the output word of the current reasoning, the information of the target phrase is integrated, and the first decoder is provided with an additional decoding reference, so that the first decoder and the second decoder can combine more information about the text to be reasoned to determine the output word, so that the output word obtained by reasoning based on the encoder, the first decoder and the second decoder is more accurate, thereby improving the accuracy of the text reasoning result. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is a structural diagram of a text reasoning system provided by an embodiment of the present application;

[0072] Figure 2 is a flowchart of a first text reasoning method provided by an embodiment of the present application;

[0073] Figure 3 is a flow chart of a second text reasoning method provided by an embodiment of the present application;

[0074] Figure 4 It is a structural diagram of an encoder, a first decoder, and a second decoder provided in an embodiment of the present application;

[0075] Figure 5 is a flowchart of a third text reasoning method provided by an embodiment of the present application;

[0076] Figure 6 is a flowchart of a fourth text reasoning method provided by an embodiment of the present application;

[0077] Figure 7 This is a flow chart of a training method for an encoder, a first decoder, and a second decoder in a text reasoning method provided in an embodiment of the present application;

[0078] Figure 8 It is a flow chart of another training method of an encoder, a first decoder, and a second decoder in a text reasoning method provided in an embodiment of the present application;

[0079] Fig. 9 This is a flow chart of a training method for an encoder and a first decoder in a text reasoning method provided in an embodiment of the present application;

[0080] Fig.10 It is a flow chart of another training method of an encoder and a first decoder in a text reasoning method provided in an embodiment of the present application;

[0081] Fig.11 is a structural schematic diagram of a text reasoning device provided by an embodiment of the present application;

[0082] Fig.12 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0083] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0084] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0085] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0086] First, the terms involved in one or more embodiments of the present invention are explained.

[0087] Encoder: The encoder is used to read the entire sentence sequence and encode it to obtain a vector representation of the sentence.

[0088] Decoder: The decoder uses the sentence vector obtained by the encoder as the target input and generates a word sequence in the target language word by word.

[0089] In the present application, a text reasoning method is provided. The present application also relates to a text reasoning device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0090] Figure 1 A structural diagram of a text reasoning system provided in one embodiment of the present application is shown.

[0091] The present application can be divided into two execution entities, namely a server and an application terminal. The server includes an encoder, a first decoder and a second decoder. The application terminal is used to provide the text to be inferred to the server, the encoder is used to encode the text to be inferred to obtain an encoding vector, the first decoder is used to decode the encoding vector and the target phrase, and the second decoder is used to decode the target phrase.

[0092] That is, the server obtains the text to be inferred from the application terminal, the encoder is used to encode the text to be inferred to obtain the encoding vector of the text to be inferred, the first decoder is used to decode the encoding vector and the target phrase to obtain a first embedding vector, the second decoder is used to decode the target phrase to obtain a second embedding vector, the first embedding vector and the second embedding vector are added, the sum is passed through the output layer to obtain the output word of the current inference, based on the output word of the current inference, the target phrase is updated, and the encoding vector and the target phrase are input into the first decoder to obtain the first embedding vector, the target phrase is input into the second decoder to obtain the second embedding vector, and when the output word of the current inference carries an end tag, the output word in the target phrase is determined to constitute the inference result of the text to be inferred, and the inference result of the text to be inferred is fed back to the application terminal.

[0093] Applying the scheme of the embodiment of the present application, the text to be inferred is obtained, the text to be inferred is input into the encoder, the encoding vector of the text to be inferred is obtained, the encoding vector and the target phrase are input into the first decoder to obtain the first embedding vector, the target phrase is input into the second decoder to obtain the second embedding vector, the first embedding vector and the second embedding vector are summed, the sum result is passed through the output layer to obtain the output word of the current inference, based on the output word of the current inference, the target phrase is updated, and the encoding vector and the target phrase are input into the first decoder to obtain the first embedding vector, the target phrase is input into the second decoder to obtain the second embedding vector, and when the output word of the current inference carries an end tag, it is determined that the output word in the target phrase constitutes the inference result of the text to be inferred. In this case, the second embedding vector is obtained by inputting the target phrase into the second decoder, the first embedding vector and the second embedding vector are summed to obtain the output word of the current inference, the information of the target phrase is integrated, and an additional decoding reference is provided to the first decoder, so that the first decoder and the second decoder can combine more information about the text to be inferred to determine the output word, so that the output word inferred based on the encoder, the first decoder and the second decoder is more accurate, thereby improving the accuracy of the text reasoning result.

[0094] Figure 2 A flowchart of a first text reasoning method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0095] Step S202: Obtain the text to be inferred, input the text to be inferred into the encoder, and obtain the encoding vector of the text to be inferred.

[0096] In the embodiment of the present application, when there is a demand for reasoning on the text, the server obtains the text to be reasoned from the terminal, wherein the text to be reasoned includes information, abstracts and other texts containing content to be reasoned, and reasoning refers to text operations including translation, analysis, extraction, and question-answering. In other words, the text to be reasoned refers to the text that can obtain the corresponding reasoning result after being processed by structures such as encoders and decoders. The text to be reasoned can be in text format or image format, and the corresponding text content is obtained through text recognition. In practical applications, the text to be reasoned can refer to text waiting to be translated, question text waiting to be answered, etc.

[0097] It should be noted that the encoder can integrate the features of the text to be inferred into the encoding vector when inferring the text to be inferred. That is, the encoding vector can represent the text to be inferred. The encoding vector is subsequently processed by the first decoder and the second decoder to obtain the inference result of the text to be inferred, thereby obtaining the inference result corresponding to the inference text simply and quickly.

[0098] For example, when the reasoning operation is a translation operation, the text to be reasoned is "I have a cat". Using the text reasoning method of the present application, the reasoning result of the text to be reasoned is "I HAVE ACAT"; when the reasoning operation is a question-and-answer operation, the text to be reasoned is "When is the winter solstice?". Using the text reasoning method of the present application, the reasoning result of the text to be reasoned is "The eighteenth day of the eleventh month of the lunar calendar".

[0099] Step S204: input the encoding vector and the target phrase into the first decoder to obtain a first embedding vector, and input the target phrase into the second decoder to obtain a second embedding vector, wherein the target phrase includes the output word that has been inferred.

[0100] In an embodiment of the present application, after the text to be inferred is input into the encoder to obtain the encoding vector of the text to be inferred, the encoding vector and the target phrase are input into the first decoder to obtain the first embedding vector, and the target phrase is input into the second decoder to obtain the second embedding vector, wherein the target phrase includes the output word that has been inferred, and in the initial case, the target phrase only includes the first character, which can be a start tag. The structures of the first decoder and the second decoder can be the same or different, and the first decoder includes but is not limited to BERT, ERNIE, etc., and the second decoder is a pre-trained language model, including but not limited to BERT, ERNIE, etc., which is selected according to the actual situation, and the embodiment of the present application does not impose any limitation on this.

[0101] It should be noted that the second decoder includes a multi-head attention layer (Multi-HeadAttention), an Add&Norm layer, a feedforward neural network layer (Feed Forward), etc., which are selected according to actual conditions, and the embodiment of the present application does not impose any limitations on this.

[0102] Specifically, in the multi-head attention layer, the query vector Query, key vector Key, and value vector Value are linearly transformed, and then input into the attention for scaling dot product operations. The parameters for linear transformation of the query vector Query, key vector Key, and value vector Value are different each time. Then the N times of scaling dot product results are concatenated, and then a linear transformation is performed to obtain the result of the multi-head attention layer. The Add&Norm layer can smoothly integrate the input and output of other layers to prevent the numerical changes within the layer from being too large, which is conducive to speeding up training and improving generalization performance. The feedforward neural network can be regarded as a function, which realizes a complex mapping from input space to output space through multiple compounding of simple nonlinear functions.

[0103] In practical applications, the number of multi-head attention layers, Add&Norm layers, and feedforward neural network layers (Feed Forward) in the first encoder and the second encoder can be the same or different, depending on the training requirements of the model. The embodiment of the present application does not impose any limitations on this.

[0104] Combined with Figure 3 , Figure 3 A flowchart of a second text reasoning method provided according to an embodiment of the present application is shown, which specifically includes:

[0105] After obtaining the text to be inferred, the text to be inferred is input into the encoder to obtain the encoding vector of the text to be inferred; the encoding vector and the target phrase are input into the first decoder to obtain the first embedding vector, and the target phrase is input into the second decoder to obtain the second embedding vector; the first embedding vector and the second embedding vector are added, and the sum is passed through the output layer to obtain the output word of the current inference; based on the output word of the current inference, the target phrase is updated, and the execution is returned to input the encoding vector and the target phrase into the first decoder to obtain the first embedding vector, and the target phrase is input into the second decoder to obtain the second embedding vector; when the output word of the current inference carries an end tag, it is determined that the output words in the target phrase constitute the inference result of the text to be inferred.

[0106] In one possible implementation, Figure 4 FIG. 4 shows a structural diagram of an encoder, a first decoder, and a second decoder provided according to an embodiment of the present application. Figure 4The input of the encoder shown in the figure can be the text to be inferred (Inputs). The word embedding (Input Embedding) and position embedding (Positional Encoding) of the text to be inferred are added together, and the query vector Query, the key vector Key, and the value vector Value are input into the multi-head attention layer (Multi-HeadAttention) and the Add&Norm layer. Then, the vectors after linear transformation are summed and normalized, and the summed and normalized vector matrices are respectively input into the feedforward neural network layer (Feed Forward) and the Add&Norm layer. Then, the vector matrices after the activation function are summed and normalized, and finally the encoding vector of the text to be inferred is obtained. The first decoder (first Decoder) can add the word Embedding (output Embedding) and position Embedding (Positional Encoding) of the target phrase (outputs) including the start tag (shifted right), input the sum and normalization of the multi-head attention layer (Masked Multi-HeadAttention) and Add&Norm layer using the Masked operation, input the summed and normalized value vector Value and the query vector Query and key vector Key of the text to be inferred output by the encoder (Encoder) into the multi-head attention layer (Multi-HeadAttention) and Add&Norm layer for summing and normalization, input the summed and normalized vector matrix into the feedforward neural network layer (Feed Forward) and Add&Norm layer for summing and normalization, and then perform linear transformation (Linear) to obtain the first embedding vector. The second decoder (second Decoder) can input the word Embedding (output Embedding) of the target phrase (outputs) into the pre-trained language model (Languagemodel) to obtain the second embedding vector. The first embedding vector and the second embedding vector are added through the softmax layer to obtain the output probability (Output Probabilities), and finally the inference result of the text to be inferred is obtained.

[0107] It should be noted that softmax is essentially a normalized network, and its purpose is to map multiple vectors into a probability distribution. Each value output by the softmax layer is in the range of (0,1). The decoder is finally a linear transformation layer and a softmax layer. The linear transformation layer is a simple fully connected neural network that can project the vector generated by the decoder into a much larger vector called logits. In practical applications, N words can be learned and an "output vocabulary" can be obtained. At this time, the log-probability vector is a vector of N cell length, and each cell corresponds to the score of a certain word. Next, the softmax layer will convert the obtained scores into probabilities. The cell with the highest probability is selected, and the word corresponding to the cell with the highest probability is used as the final output result.

[0108] Step S206: Add the first embedding vector and the second embedding vector, and pass the sum result through the output layer to obtain the output word of the current reasoning.

[0109] In an embodiment of the present application, the encoding vector and the target phrase are input into the first decoder to obtain the first embedding vector, the target phrase is input into the second decoder to obtain the second embedding vector, and then the first embedding vector and the second embedding vector are added, and the sum is passed through the output layer to obtain the output word of the current reasoning. Among them, the first embedding vector can represent the encoding vector and the target phrase, and the second embedding vector can represent the target phrase. On the basis that the encoding vector represents the text to be reasoned, the first embedding vector and the second embedding vector are added, which not only considers the text to be reasoned, but also considers the target phrase, that is, the accuracy of the output word of the current reasoning determined based on the first embedding vector and the second embedding vector will be higher.

[0110] Step S208: Based on the output word of the current inference, update the target phrase and return to execute step S204.

[0111] In an embodiment of the present application, after obtaining the output word of the current reasoning, the target phrase can be updated based on the output word of the current reasoning. The method of updating the target phrase includes adding the output word of the current reasoning as a new character to the target phrase, thereby returning to execute inputting the encoding vector and the target phrase into the first decoder to obtain a first embedding vector, inputting the target phrase into the second decoder to obtain a second embedding vector, and completing the reasoning of the text to be reasoned.

[0112] In the embodiment of the present application, since the first decoder decodes the encoding vector and the target phrase once, the second decoder also decodes the target phrase in sequence according to word units, and each word unit may include only one character or multiple characters, after obtaining the output word of the current reasoning and updating the target phrase, the reasoning result of the entire text to be reasoned is not directly obtained, so it is possible to return to continue decoding the remaining word units that have not been decoded, until the output word of the current reasoning carries an end tag, it is determined that the text to be reasoned has been reasoned and the final reasoning result of the text to be reasoned is obtained.

[0113] Step S210: when the output word of the current reasoning carries an end tag, determine the reasoning result of the output word in the target phrase constituting the text to be reasoned.

[0114] In an embodiment of the present application, after updating the target phrase, it can be determined whether the output word of the current reasoning carries an end tag, thereby determining whether the current reasoning has obtained the reasoning result of the text to be reasoned. The order of determining whether the output word of the current reasoning carries an end tag and updating the target phrase is not limited. One method can be: based on the output word of the current reasoning, after updating the target phrase, determine whether the output word of the current reasoning carries an end tag. If the output word of the current reasoning does not carry an end tag, return to execute the input of the encoding vector and the target phrase into the first decoder to obtain the first embedding vector, and input the target phrase into the second decoder to obtain the second embedding vector; based on the output word of the current reasoning, after updating the target phrase, determine whether the output word of the current reasoning carries an end tag. If the output word of the current reasoning carries an end tag, determine that the output word in the target phrase constitutes the reasoning result of the text to be reasoned.

[0115] Another way can be: determine whether the output word currently inferred carries an end tag. If the output word currently inferred does not carry an end tag, update the target phrase based on the output word currently inferred, return to execute, input the encoding vector and the target phrase into the first decoder, obtain the first embedding vector, input the target phrase into the second decoder, and obtain the second embedding vector; if the output word currently inferred carries an end tag, it means that the text to be inferred has been inferred, update the target phrase based on the output word currently inferred, and determine that the output words in the target phrase constitute the inference result of the text to be inferred.

[0116] Apply the scheme of the embodiment of the present application, obtain the text to be inferred, input the text to be inferred into the encoder, obtain the encoding vector of the text to be inferred, input the encoding vector and the target phrase into the first decoder, obtain the first embedding vector, input the target phrase into the second decoder, obtain the second embedding vector, sum the first embedding vector and the second embedding vector, obtain the output word of the current inference through the output layer, update the target phrase based on the output word of the current inference, return to execute, input the encoding vector and the target phrase into the first decoder, obtain the first embedding vector, input the target phrase into the second decoder, obtain the second embedding vector, and in the case where the output word of the current inference carries an end tag, confirm The output word in the target phrase constitutes the reasoning result of the text to be inferred. In this case, the target phrase is input into the second decoder to obtain a second embedding vector, and the first embedding vector and the second embedding vector are summed to obtain the output word of the current reasoning, which incorporates the information of the target phrase and provides an additional decoding reference to the first decoder. When the first decoder and the second decoder are different, the functions of the two decoders can be combined, so that the first decoder and the second decoder can combine more information about the text to be inferred to determine the output word, so that the output word obtained by reasoning based on the encoder, the first decoder and the second decoder is more accurate, thereby improving the accuracy of the text reasoning result.

[0117] based on Figure 2 In the embodiment shown, step S206 can be specifically performed as follows: Figure 5 The flowchart shown implements, Figure 5 A flowchart of a third text reasoning method provided in an embodiment of the present application is shown, and the method specifically comprises the following steps:

[0118] Step S502: Obtain a first weight coefficient preset for the first decoder and a second weight coefficient preset for the second decoder.

[0119] Step S504: weighting the first embedding vector and the second embedding vector using the first weight coefficient and the second weight coefficient, and obtaining the output word of the current reasoning through the output layer.

[0120] In an embodiment of the present application, in order to enable the decoder to obtain a more accurate decoding result, weight coefficients are pre-set for the decoders, the first decoder pre-sets a first weight coefficient, and the second decoder pre-sets a second weight coefficient, wherein the weight coefficients are generally obtained through training, and can also be set artificially based on experience. The specific selection is based on actual conditions, and the embodiment of the present application does not impose any limitation on this.

[0121] Specifically, the first weight coefficient may be multiplied by the first embedding vector to obtain a first result, the second weight coefficient may be multiplied by the second embedding vector to obtain a second result, and the first result and the second result may be added to obtain a weighted result.

[0122] One possible implementation method is that when training the weight coefficients, the first weight coefficient can be set to 0.4. When the second weight coefficient is set to 0.6, the inference result obtained is more accurate than the inference result when the first weight coefficient and the second weight coefficient are both 0.5. Therefore, in order to improve the accuracy of the inference result corresponding to the text to be inferred, the second weight coefficient can be continued to be increased and the first weight coefficient can be reduced until the first weight coefficient and the second weight coefficient that make the inference result most accurate are found.

[0123] In an embodiment of the present application, a first weight coefficient is pre-set for the first decoder, and a second weight coefficient is pre-set for the second decoder, the sum of the first weight coefficient and the second weight coefficient is 1, the first weight coefficient can determine the influence of the first decoder on the entire reasoning process, and the second weight coefficient can determine the influence of the second decoder on the entire reasoning process, so by setting the first weight coefficient and the second weight coefficient, the accuracy of the encoder, the first decoder and the second decoder can be adjusted, thereby improving the accuracy of the reasoning results corresponding to the text to be inferred.

[0124] Figure 6 A flowchart of a fourth text reasoning method provided in an embodiment of the present application is shown, and the method specifically comprises the following steps:

[0125] Step S602: Obtain the text to be inferred, input the text to be inferred into the encoder, and obtain the encoding vector of the text to be inferred.

[0126] Step S604: input the encoding vector and the start tag into the first decoder to obtain a first embedding vector, and input the start tag into the second decoder to obtain a second embedding vector.

[0127] Step S606: Add the first embedding vector and the second embedding vector, and pass the sum result through the output layer to obtain the output word of the current reasoning.

[0128] Step S608: Add the output word of the current inference as a new character to the target phrase, and return to execute step S604.

[0129] Step S610: When the output word currently inferred carries an end tag, the output words in the target phrase are merged in the order in which they are inferred, to obtain the inference result of the text to be inferred.

[0130] In the embodiment of the present application, text reasoning is performed word by word according to the order of words in the text to be reasoned. Therefore, the inferred output words also have an order of order. By merging them according to the inferred order of order, the reasoning result of the text to be reasoned can be obtained.

[0131] For example, get the text to be inferred "I love music", encode it in the encoder, get the encoding vector of the text to be inferred "I love music", and start label " <begin>" and the encoding vector are merged and input into the first decoder in the order of the start tag first and the encoding vector last to obtain the first embedding vector, the start tag is input into the second decoder to obtain the second embedding vector, the first embedding vector and the second embedding vector are added, and the sum is passed through the output layer to obtain the output word "I" of the current reasoning, and it is determined that the current output word "I" does not carry the end tag, and "I" is added to the target phrase. At this time, the updated target phrase includes the start tag" <begin>” and “I”.

[0132] Will start the tag " <begin>" and "I" and the encoding vector are input into the first decoder to obtain the first embedding vector, the start tag is input into the second decoder to obtain the second embedding vector, the first embedding vector and the second embedding vector are summed, and the sum is passed through the output layer to obtain the current inference output word "LOVE", and it is determined that the current output word "LOVE" does not carry an end tag, and "LOVE" is added to the target phrase. At this time, the updated target phrase includes the start tag " <begin>”, “I” and “LOVE”.

[0133] Will start the tag " <begin>”, "I" and "LOVE" and the encoding vector are input into the first decoder to obtain the first embedding vector, the start tag is input into the second decoder to obtain the second embedding vector, the first embedding vector and the second embedding vector are summed, and the sum result is passed through the output layer to obtain the output word "MUSIC" of the current reasoning, and "MUSIC" is added to the target phrase. At this time, it is determined that the output word "MUSIC" of the current reasoning carries the end tag, and the output words "I", "LOVE", and "MUSIC" in the target phrase are merged in the order in which they are inferred, and the reasoning result "I LOVEMUSIC" of the text to be inferred is obtained.

[0134] By applying the scheme of the embodiment of the present application, after obtaining the encoding vector of the text to be inferred, the encoding vector and the start tag are input into the first decoder to obtain the first embedding vector, the start tag is input into the second decoder to obtain the second embedding vector, wherein the start tag is the first character in the target phrase, the first embedding vector and the second embedding vector are summed, and the sum result is passed through the output layer to obtain the output word of the current inference, and the output word of the current inference is added as a new character to the target phrase, and the execution of step S604 is returned. When the output word of the current inference carries the end tag, the output words in the target phrase are merged in the order in which they are inferred to obtain the inference result of the text to be inferred. In this case, the second embedding vector is obtained by inputting the target phrase into the second decoder, and the first embedding vector and the second embedding vector are summed to obtain the output word of the current inference, which incorporates the information of the target phrase, and provides an additional decoding reference to the first decoder, so that the first decoder and the second decoder can combine more information about the text to be inferred to determine the output word, so that the output word inferred based on the encoder, the first decoder and the second decoder is more accurate, thereby improving the accuracy of the text reasoning result.

[0135] Figure 7 A flow chart of a training method for an encoder, a first decoder, and a second decoder in a text reasoning method provided in an embodiment of the present application is shown, which specifically includes the following steps:

[0136] Step S702: obtaining a sample set, wherein the sample set includes a plurality of sample texts and target texts corresponding to the sample texts.

[0137] In the embodiment of the present application, when training the encoder, the first decoder and the second decoder, it is necessary to obtain a sample set including multiple sample texts and target texts corresponding to each sample text. For example, the sample text is "I have a cat", and the target text corresponding to the sample text is "I HAVEA CAT". Generally, the method of obtaining the sample set can be to form a sample set with a large number of sample texts input manually, or to form a sample set with a large number of sample texts read from other data acquisition devices or databases. The method of obtaining the sample set is selected according to the actual situation, and the embodiment of the present application does not impose any limitation on this.

[0138] Step S704: extract the first sample text, input the first sample text into the encoder, and obtain the first sample encoding vector of the first sample text.

[0139] Step S706: Input the first sample encoding vector and the first target phrase into the first decoder to obtain a first sample embedding vector, and input the first target phrase into the second decoder to obtain a second sample embedding vector, wherein the first target phrase includes the output word that has been inferred for the first sample text.

[0140] Step S708: Add the first sample embedding vector and the second sample embedding vector, and pass the sum result through the output layer to obtain the first current output word.

[0141] Step S710: Based on the first current output word, update the first target phrase, and return to execute step S706.

[0142] Step S712: when the first current output word carries an end tag, determine a first inference result that the output words in the first target phrase constitute the first sample text.

[0143] Step S714: According to the first inference result and the first target text corresponding to the first sample text, the encoder, the first decoder and the second decoder are trained to obtain the trained encoder, the first decoder and the second decoder.

[0144] In an embodiment of the present application, after obtaining the first reasoning result, the encoder, the first decoder, and the second decoder can be trained according to the first reasoning result and the first target text corresponding to the first sample text to obtain the trained encoder, the first decoder, and the second decoder, so that the trained encoder, the first decoder, and the second decoder are more accurate.

[0145] In a possible implementation, the above step S714 is implemented as follows: Figure 8 As shown, Figure 8 A flowchart of another training method of an encoder, a first decoder, and a second decoder in a text reasoning method provided in an embodiment of the present application is shown, which may specifically include the following steps:

[0146] Step S802: Calculate a first difference value between the first inference result and the first target text corresponding to the first sample text.

[0147] Step S804: If the first difference value is greater than the first preset threshold, adjust the model parameters of the encoder, the first decoder and the second decoder, and return to execute the extraction of the first sample text, input the first sample text into the encoder, and obtain the first sample encoding vector of the first sample text.

[0148] Step S806: If the first difference value is less than or equal to the first preset threshold, the training is stopped to obtain the trained encoder, the first decoder and the second decoder.

[0149] In an embodiment of the present application, the second decoder can be trained synchronously with the encoder and the first decoder. After obtaining the first inference result, the second decoder compares the first inference result with the first target text corresponding to the first sample text, calculates the first difference value, and compares the first difference value with the first preset threshold.

[0150] Specifically, if the first difference value is greater than the first preset threshold, it means that the difference between the first inference result and the first target text corresponding to the first sample text is large, and the recognition and reasoning ability of the encoder, the first decoder and the second decoder for the text to be inferred is poor. At this time, the model parameters of the encoder, the first decoder and the second decoder can be adjusted, and the execution of extracting the first sample text is returned, and the first sample text is input into the encoder to obtain the first sample encoding vector of the first sample text. The encoder, the first decoder and the second decoder are continued to be trained until the first difference value is less than or equal to the first preset threshold, indicating that the difference between the first inference result and the first target text corresponding to the first sample text is small, and the training is completed to obtain the trained encoder, the first decoder and the second decoder.

[0151] By applying the solution of the embodiment of the present application, a first difference value between the first inference result and the first target text corresponding to the first sample text is calculated, and the first difference value is compared with a first preset threshold value. When the difference value is greater than the first preset threshold value, the encoder, the first decoder and the second decoder are continued to be trained until the training is completed when the difference value is less than or equal to the first preset threshold value. By continuously adjusting the model parameters of the encoder, the first decoder and the second decoder, the final encoder, the first decoder and the second decoder can be made more accurate, thereby improving the accuracy of the inference results.

[0152] In another possible implementation, in addition to comparing the size relationship between the first difference value and the first preset threshold, the implementation of the above step S714 can also combine the number of iterations to determine whether the current encoder, first decoder and second decoder have been trained. Specifically, if the first difference value is greater than the first preset threshold, the model parameters of the encoder, the first decoder and the second decoder are adjusted, and the first sample text is extracted and input into the encoder to obtain the first sample encoding vector of the first sample text. The encoder, the first decoder and the second decoder are continuously trained until the preset number of iterations is reached, and the iteration is stopped to obtain the trained encoder, the first decoder and the second decoder, wherein the first preset threshold and the preset number of iterations are selected according to the actual situation, and the embodiment of the present application does not impose any limitation on this.

[0153] In practical applications, the first difference value refers to the loss value. There are many functions for calculating the first difference value, such as cross entropy loss function, L1 norm loss function, maximum loss function, mean square error loss function, logarithmic loss function, etc. In this application, there is no limitation on the choice of function for calculating the difference value.

[0154] Preferably, a cross entropy loss function may be used to calculate the cross entropy between the first inference result and the first target text corresponding to the first sample text as the first difference value, wherein the cross entropy loss function is:

[0155]

[0156] Among them, C represents the number of categories, p i is the true probability value, q i is the predicted probability value, and i is the i-th word.

[0157] By using the cross entropy loss function, the cross entropy between the first inference result and the first target text corresponding to the first sample text is calculated as the first difference value, which improves the efficiency of calculating the first difference value and thus improves the accuracy of the text reasoning result.

[0158] By applying the solution of the embodiment of the present application, a first difference value between the first inference result and the first target text corresponding to the first sample text is calculated, and the first difference value is compared with a first preset threshold value. When the difference value is greater than the first preset threshold value, the encoder, the first decoder and the second decoder are continued to be trained until the training is completed when a preset number of iterations is reached. By continuously adjusting the model parameters of the encoder, the first decoder and the second decoder, the final encoder, the first decoder and the second decoder can be made more accurate, thereby improving the accuracy of the inference results.

[0159] Fig. 9 A flow chart of a training method for an encoder and a first decoder in a text reasoning method provided in an embodiment of the present application is shown, which specifically includes the following steps:

[0160] Step S902: Obtain a sample set, which includes a plurality of sample texts and target texts corresponding to the sample texts.

[0161] Step S904: extract the second sample text, input the second sample text into the encoder, and obtain a second sample encoding vector of the second sample text.

[0162] Step S906: Input the second sample encoding vector and the second target phrase into the first decoder to obtain a third sample embedding vector, and input the second target phrase into the second decoder to obtain a fourth sample embedding vector, wherein the second target phrase includes the output word that has been inferred for the second sample text.

[0163] Step S908: Add the third sample embedding vector and the fourth sample embedding vector, and pass the sum result through the output layer to obtain the second current output word.

[0164] Step S910: Based on the second current output word, update the second target phrase, and return to execute step S906.

[0165] Step S912: when the second current output word carries an end tag, determine a second inference result that the output words in the second target phrase constitute a second sample text.

[0166] Step S914: According to the second inference result and the second target text corresponding to the second sample text, the encoder and the first decoder are trained to obtain the trained encoder and the first decoder.

[0167] In an embodiment of the present application, after obtaining the second reasoning result, the encoder and the first decoder can be trained according to the second reasoning result and the second target text corresponding to the second sample text to obtain the trained encoder and the first decoder, so that the trained encoder and the first decoder are more accurate. By using the pre-trained second encoder and giving the first decoder as a reference when training the first decoder, the training efficiency of the first decoder is improved, thereby improving the efficiency of text reasoning.

[0168] In a possible implementation, the above step S914 is implemented as follows: Fig.10 As shown, Fig.10 A flowchart of another training method for an encoder and a first decoder in another text reasoning method provided in an embodiment of the present application is shown, which may specifically include the following steps:

[0169] Step S1002: Calculate a second difference value between the second inference result and the second target text corresponding to the second sample text.

[0170] Step S1004: If the second difference value is greater than the second preset threshold, the model parameters of the encoder and the first decoder are adjusted, the model parameters of the second decoder are kept unchanged, and the execution is returned to obtain a sample set, which includes multiple sample texts and target texts corresponding to each sample text.

[0171] Step S1006: If the second difference value is less than or equal to the second preset threshold, stop the training and obtain the trained encoder and the first decoder.

[0172] In the embodiment of the present application, the second decoder can be pre-trained, and the pre-trained second decoder can be directly obtained. During the training process, only the encoder and the first decoder are trained. The encoder and the first decoder are trained by the above training method. After obtaining the second inference result, the second inference result is compared with the second target text corresponding to the second sample text, and the second difference value is calculated, and the second difference value is compared with the second preset threshold.

[0173] Specifically, if the second difference value is greater than the second preset threshold, it means that the difference between the second inference result and the second target text corresponding to the second sample text is large, and the recognition and reasoning ability of the encoder and the first decoder for the text to be inferred is poor. At this time, the model parameters of the encoder and the first decoder can be adjusted, and the execution is returned to obtain the sample set, which includes multiple sample texts and target texts corresponding to each sample text. The encoder and the first decoder continue to be trained until the second difference value is less than or equal to the second preset threshold, indicating that the difference between the second inference result and the second target text corresponding to the second sample text is small, and the training is completed to obtain the trained encoder and first decoder.

[0174] By applying the scheme of the embodiment of the present application, a second difference value between the second inference result and the second target text corresponding to the second sample text is calculated, the second difference value is compared with a second preset threshold, and the encoder and the first decoder are continued to be trained when the difference is greater than the second preset threshold until the training is completed when the difference is less than or equal to the first preset threshold. By continuously adjusting the model parameters of the encoder and the first decoder, the final encoder and the first decoder can be made more accurate, thereby improving the accuracy of the inference results. By using the pre-trained second encoder and giving the first decoder as a reference when training the first decoder, the training efficiency of the first decoder is improved, thereby improving the efficiency of text reasoning.

[0175] In another possible implementation, in addition to comparing the magnitude relationship between the second difference value and the second preset threshold, the implementation of the above step S914 can also combine the number of iterations to determine whether the current encoder and the first decoder have been trained. Specifically, if the second difference value is greater than the second preset threshold, the model parameters of the encoder and the first decoder are adjusted, and the execution returns to obtain a sample set, which includes multiple sample texts and target texts corresponding to each sample text, and the encoder and the first decoder are continuously trained until the preset number of iterations is reached, and the iteration is stopped to obtain the trained encoder and the first decoder, wherein the first preset threshold and the preset number of iterations are selected according to the actual situation, and the embodiment of the present application does not impose any limitation on this.

[0176] By applying the scheme of the embodiment of the present application, a second difference value between the second inference result and the second target text corresponding to the second sample text is calculated, the second difference value is compared with a second preset threshold value, and when the difference value is greater than the second preset threshold value, the encoder and the first decoder are continued to be trained until the training is completed when a preset number of iterations is reached. By continuously adjusting the model parameters of the encoder and the first decoder, the final encoder and the first decoder can be made more accurate, and the accuracy of the inference results is improved. By using the pre-trained second encoder and giving the first decoder as a reference when training the first decoder, the training efficiency of the first decoder is improved, thereby improving the efficiency of text reasoning.

[0177] Corresponding to the above method embodiment, the present application also provides a text reasoning device embodiment, Fig.11 FIG. 1 shows a schematic diagram of the structure of a text reasoning device provided by an embodiment of the present application. Fig.11 As shown, the device comprises:

[0178] The encoding vector obtaining module 1102 is configured to obtain the text to be inferred, input the text to be inferred into the encoder, and obtain the encoding vector of the text to be inferred;

[0179] The embedding vector obtaining module 1104 is configured to input the encoding vector and the target phrase into a first decoder to obtain a first embedding vector, and input the target phrase into a second decoder to obtain a second embedding vector, wherein the target phrase includes the output word that has been inferred;

[0180] An output word obtaining module 1106 is configured to add the first embedding vector and the second embedding vector, and obtain an output word of current reasoning through an output layer by passing the sum result;

[0181] An updating module 1108 is configured to update a target phrase based on the output word of the current reasoning, return to execute inputting the encoding vector and the target phrase into the first decoder to obtain a first embedding vector, and inputting the target phrase into the second decoder to obtain a second embedding vector;

[0182] The inference result determination module 1110 is configured to determine the inference result of the output word in the target phrase constituting the text to be inferred when the output word of the current inference carries an end tag.

[0183] Optionally, the second decoder is a pre-trained language model.

[0184] Optionally, the first character in the target phrase is a start tag;

[0185] In the initial case, the embedding vector obtaining module 1104 is further configured to input the encoding vector and the start tag into the first decoder to obtain the first embedding vector, and input the start tag into the second decoder to obtain the second embedding vector;

[0186] The updating module 1108 is further configured to add the output word of the current reasoning as a new character to the target word group;

[0187] The inference result determination module 1110 is further configured to merge the output words in the target phrase according to the inferred sequence to obtain the inference result of the text to be inferred.

[0188] Optionally, the output word acquisition module 1106 is further configured to obtain a first weight coefficient pre-set for the first decoder and a second weight coefficient pre-set for the second decoder; use the first weight coefficient and the second weight coefficient to weight the first embedding vector and the second embedding vector, and obtain the output word of the current inference through the output layer.

[0189] Optionally, the device also includes: a first training module, configured to obtain a sample set, the sample set includes multiple sample texts and target texts corresponding to each sample text; extract the first sample text, input the first sample text into the encoder, and obtain a first sample encoding vector of the first sample text; input the first sample encoding vector and the first target phrase into the first decoder to obtain a first sample embedding vector, input the first target phrase into the second decoder to obtain a second sample embedding vector, wherein the first target phrase includes an output word that has been inferred for the first sample text; sum the first sample embedding vector and the second sample embedding vector, and obtain the first current output word through the output layer; based on the first current output word, update the first target phrase, return to execute inputting the first sample encoding vector and the first target phrase into the first decoder to obtain the first sample embedding vector, inputting the first target phrase into the second decoder to obtain the second sample embedding vector; when the first current output word carries an end tag, determine that the output words in the first target phrase constitute the first inference result of the first sample text; according to the first inference result and the first target text corresponding to the first sample text, train the encoder, the first decoder and the second decoder to obtain the trained encoder, the first decoder and the second decoder.

[0190] Optionally, the first training module is further configured to calculate a first difference value between the first inference result and the first target text corresponding to the first sample text; if the first difference value is greater than a first preset threshold, adjust the model parameters of the encoder, the first decoder and the second decoder, and return to execute the extraction of the first sample text, input the first sample text into the encoder, and obtain a first sample encoding vector of the first sample text; if the first difference value is less than or equal to the first preset threshold, stop training and obtain the trained encoder, first decoder and second decoder.

[0191] Optionally, the second decoder is pre-trained;

[0192] The device also includes: a second training module, which is configured to obtain a sample set, which includes multiple sample texts and target texts corresponding to each sample text; extract the second sample text, input the second sample text into the encoder, and obtain a second sample encoding vector of the second sample text; input the second sample encoding vector and the second target phrase into the first decoder to obtain a third sample embedding vector, input the second target phrase into the second decoder, and obtain a fourth sample embedding vector, wherein the second target phrase includes an output word that has been inferred for the second sample text; sum the third sample embedding vector and the fourth sample embedding vector, and obtain a second current output word through the output layer; based on the second current output word, update the second target phrase, return to execute inputting the second sample encoding vector and the second target phrase into the first decoder to obtain the third sample embedding vector, inputting the second target phrase into the second decoder to obtain the fourth sample embedding vector; in the case where the second current output word carries an end tag, determine that the output words in the second target phrase constitute a second inference result of the second sample text; according to the second inference result and the second target text corresponding to the second sample text, train the encoder and the first decoder to obtain the trained encoder and the first decoder.

[0193] Optionally, the second training module is further configured to calculate a second difference value between the second inference result and the second target text corresponding to the second sample text; if the second difference value is greater than a second preset threshold, the model parameters of the encoder and the first decoder are adjusted, the model parameters of the second decoder are kept unchanged, and the second sample text is extracted and input into the encoder to obtain a second sample encoding vector of the second sample text; if the second difference value is less than or equal to the second preset threshold, the training is stopped to obtain the trained encoder and first decoder.

[0194] Applying the scheme of the embodiment of this specification, the text to be inferred is obtained, the text to be inferred is input into the encoder, the encoding vector of the text to be inferred is obtained, the encoding vector and the target phrase are input into the first decoder to obtain the first embedding vector, the target phrase is input into the second decoder to obtain the second embedding vector, the first embedding vector and the second embedding vector are summed, the sum result is passed through the output layer to obtain the output word of the current inference, based on the output word of the current inference, the target phrase is updated, and the encoding vector and the target phrase are input into the first decoder to obtain the first embedding vector, the target phrase is input into the second decoder to obtain the second embedding vector, and when the output word of the current inference carries an end tag, it is determined that the output word in the target phrase constitutes the inference result of the text to be inferred. In this case, the second embedding vector is obtained by inputting the target phrase into the second decoder, the first embedding vector and the second embedding vector are summed to obtain the output word of the current inference, the information of the target phrase is integrated, and the first decoder is provided with an additional decoding reference, so that the first decoder and the second decoder can combine more information about the text to be inferred to determine the output word, so that the output word inferred based on the encoder, the first decoder and the second decoder is more accurate, thereby improving the accuracy of the text reasoning result.

[0195] The above is a schematic scheme of a text reasoning device of the present embodiment. It should be noted that the technical scheme of the text reasoning device and the technical scheme of the above-mentioned text reasoning method belong to the same concept. For the details not described in detail in the technical scheme of the text reasoning device, please refer to the description of the technical scheme of the above-mentioned text reasoning method. In addition, each component in the device embodiment should be understood as a functional module that must be established to implement each step of the program flow or each step of the method, and each functional module is not an actual functional division or separation definition. The device claim defined by such a group of functional modules should be understood as a functional module architecture that mainly implements the solution through the computer program recorded in the specification, and should not be understood as a physical device that mainly implements the solution through hardware.

[0196] Fig.12 The structure block diagram of a computing device 1200 provided according to an embodiment of the present application is shown. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0197] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0198] In one embodiment of the present application, the above components of the computing device 1200 and Fig.12 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0199] The computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 1200 may also be a mobile or stationary server.

[0200] The processor 1220 is used to execute computer executable instructions of the text reasoning method.

[0201] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned text reasoning method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned text reasoning method.

[0202] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions, which are used for a text reasoning method when executed by a processor.

[0203] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned text reasoning method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned text reasoning method.

[0204] An embodiment of the present application discloses a chip storing computer instructions, which implement the steps of the text reasoning method as described above when executed by a processor.

[0205] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0206] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0207] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0208] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0209] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.< / begin> < / begin> < / begin> < / begin> < / begin>

Claims

1. A text reasoning method, characterized in that: include: Obtaining a text to be inferred, inputting the text to be inferred into an encoder, and obtaining an encoding vector of the text to be inferred; Inputting the encoding vector and the target word group into a first decoder to obtain a first embedding vector, and inputting the target word group into a second decoder to obtain a second embedding vector, wherein the target word group includes the inferred output word, and the structures of the first decoder and the second decoder are the same or different; Adding the first embedding vector and the second embedding vector, and passing the sum result through the output layer to obtain the output word of the current reasoning; Based on the output word of the current reasoning, the target word group is updated, and the step of inputting the encoding vector and the target word group into the first decoder to obtain a first embedding vector is returned to be executed, and the target word group is input into the second decoder to obtain a second embedding vector; In the case where the output word of the current reasoning carries an end tag, it is determined that the output words in the target phrase constitute the reasoning result of the text to be reasoned.

2. The method according to claim 1, characterized in that The second decoder is a pre-trained language model.

3. The method according to claim 1, characterized in that The first character in the target phrase is the start tag; In an initial situation, the step of inputting the encoding vector and the target phrase into a first decoder to obtain a first embedding vector, and inputting the target phrase into a second decoder to obtain a second embedding vector comprises: Input the encoding vector and the start tag into the first decoder to obtain a first embedding vector, and input the start tag into the second decoder to obtain a second embedding vector; The step of updating the target phrase based on the output word of the current reasoning includes: Add the output word of the current reasoning as a new character to the target phrase; The step of determining that the output words in the target phrase constitute the reasoning result of the text to be inferred includes: The output words in the target phrase are merged according to the order in which they are inferred to obtain the inference result of the text to be inferred.

4. The method according to any one of claims 1 to 3, characterized in that: The step of adding the first embedding vector and the second embedding vector and obtaining the output word of the current reasoning through the output layer by adding the sum includes: Acquire a first weight coefficient preset for the first decoder and a second weight coefficient preset for the second decoder; The first weight coefficient and the second weight coefficient are used to weight the first embedding vector and the second embedding vector, and the weighted result is passed through the output layer to obtain the output word of the current reasoning.

5. The method according to any one of claims 1 to 3, characterized in that: The training method of the encoder, the first decoder and the second decoder includes: Acquire a sample set, wherein the sample set includes a plurality of sample texts and target texts corresponding to the sample texts; Extracting a first sample text, inputting the first sample text into the encoder, and obtaining a first sample encoding vector of the first sample text; Inputting the first sample encoding vector and the first target phrase into a first decoder to obtain a first sample embedding vector, and inputting the first target phrase into a second decoder to obtain a second sample embedding vector, wherein the first target phrase includes an output word that has been inferred for the first sample text; Adding the first sample embedding vector and the second sample embedding vector, and passing the sum result through an output layer to obtain a first current output word; Based on the first current output word, the first target phrase is updated, and the first sample encoding vector and the first target phrase are input into the first decoder to obtain a first sample embedding vector, and the first target phrase is input into the second decoder to obtain a second sample embedding vector; In the case where the first current output word carries an end tag, determining that the output words in the first target phrase constitute a first inference result of the first sample text; According to the first inference result and the first target text corresponding to the first sample text, the encoder, the first decoder and the second decoder are trained to obtain the trained encoder, the first decoder and the second decoder.

6. The method according to claim 5, characterized in that The step of training the encoder, the first decoder, and the second decoder according to the first inference result and the first target text corresponding to the first sample text to obtain the trained encoder, the first decoder, and the second decoder includes: Calculating a first difference value between the first inference result and a first target text corresponding to the first sample text; If the first difference value is greater than a first preset threshold, adjusting the model parameters of the encoder, the first decoder and the second decoder, and returning to execute the extraction of the first sample text, inputting the first sample text into the encoder, and obtaining a first sample encoding vector of the first sample text; If the first difference value is less than or equal to the first preset threshold, the training is stopped to obtain the trained encoder, the first decoder and the second decoder.

7. The method according to claim 1, characterized in that The second decoder has been pre-trained; The training method of the encoder and the first decoder includes: Acquire a sample set, wherein the sample set includes a plurality of sample texts and target texts corresponding to the sample texts; Extracting a second sample text, inputting the second sample text into the encoder, and obtaining a second sample encoding vector of the second sample text; Inputting the second sample encoding vector and the second target phrase into the first decoder to obtain a third sample embedding vector, and inputting the second target phrase into the second decoder to obtain a fourth sample embedding vector, wherein the second target phrase includes an output word that has been inferred for the second sample text; Adding the third sample embedding vector and the fourth sample embedding vector, and passing the sum result through the output layer to obtain a second current output word; Based on the second current output word, updating the second target phrase, returning to execute the step of inputting the second sample encoding vector and the second target phrase into the first decoder to obtain a third sample embedding vector, inputting the second target phrase into the second decoder to obtain a fourth sample embedding vector; In the case where the second current output word carries an end tag, determining that the output words in the second target phrase constitute a second inference result of the second sample text; According to the second inference result and the second target text corresponding to the second sample text, the encoder and the first decoder are trained to obtain the trained encoder and the first decoder.

8. The method according to claim 7, characterized in that The step of training the encoder and the first decoder according to the second inference result and the second target text corresponding to the second sample text to obtain the trained encoder and the first decoder includes: Calculate a second difference value between the second inference result and the second target text corresponding to the second sample text; If the second difference value is greater than a second preset threshold, adjusting the model parameters of the encoder and the first decoder, keeping the model parameters of the second decoder unchanged, returning to execute the extraction of the second sample text, inputting the second sample text into the encoder, and obtaining a second sample encoding vector of the second sample text; If the second difference value is less than or equal to the second preset threshold, the training is stopped to obtain the trained encoder and the first decoder.

9. A text reasoning device, characterized in that: include: The encoding vector obtaining module is configured to obtain the text to be inferred, input the text to be inferred into the encoder, and obtain the encoding vector of the text to be inferred; an embedding vector obtaining module, configured to input the encoding vector and the target phrase into a first decoder to obtain a first embedding vector, and input the target phrase into a second decoder to obtain a second embedding vector, wherein the target phrase includes an output word that has been inferred, and the structures of the first decoder and the second decoder are the same or different; An output word obtaining module is configured to add the first embedding vector and the second embedding vector, and obtain an output word of current reasoning through an output layer by passing the sum result; An updating module is configured to update the target phrase based on the output word of the current reasoning, return to execute the step of inputting the encoding vector and the target phrase into the first decoder to obtain a first embedding vector, and inputting the target phrase into the second decoder to obtain a second embedding vector; The inference result determination module is configured to determine, when the output word of the current inference carries an end tag, that the output words in the target phrase constitute the inference result of the text to be inferred.

10. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the text reasoning method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the text reasoning method described in any one of claims 1 to 8 are implemented.

Images