A text generation method, device, computer device, and storage medium

Abstract

An embodiment of the present disclosure discloses a text generation method, apparatus, computer device, and storage medium. The method includes: inputting the obtained original text into a trained text encoding model to obtain text feature information; based on the text feature information, combining with a trained text decoding model to generate a target text corresponding to the original text; the text decoding model includes a text prediction layer, the node information of a set number of nodes included in the text prediction layer is determined by the text feature information, and the target words included in the target text and the combination order of each target word are determined by the node information of each of the nodes and the topological structure between the nodes. Using this method, parallel determination of the node information of the nodes and parallel determination of each word in the generated text are realized, reducing the text generation delay, while better avoiding the occurrence of consecutive repeated words in the generated text, ensuring the context relevance in the generated text, and improving the generation quality of the generated text.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of natural language processing, and in particular, to a text generation method, apparatus, computer device, and storage medium. Background Art

[0002] Text generation technology is an important technology in the field of natural language processing. Through text generation technology, a text sequence that meets specific goals can be generated using established information and a text generation model. Among them, after the text generation model used is trained based on sample data in different application scenarios (generative reading comprehension, human-computer dialogue, intelligent writing, machine translation, etc.), text generation in different application scenarios can be achieved.

[0003] Currently, one problem with the text generation model used in text generation implementation is that there is a relatively high output delay during text generation (output delay refers to the time delay required for the model to receive input until the model completely generates text output). And this output delay is linearly related to the sentence length of the generated text. Or, when solving the output delay problem, new problems will be introduced, such as the generated text may have consecutive repeated words or the context may be incoherent. Summary of the Invention

[0004] Embodiments of the present disclosure provide a text generation method, apparatus, computer device, and storage medium, which reduce the incoherence of the context of the generated text and consecutive repeated words, and improve the quality of the generated text.

[0005] In a first aspect, embodiments of the present disclosure provide a text generation method, which includes:

[0006] Input the obtained original text into a trained text encoding model to obtain text feature information;

[0007] Based on the text feature information, combine with a trained text decoding model to generate a target text corresponding to the original text;

[0008] Wherein, the text decoding model includes a text prediction layer, the node information of a set number of nodes included in the text prediction layer is determined based on the text feature information, and the target words included in the target text and the combination order of each target word are determined by the node information of each node and the topological structure between nodes.

[0009] In a second aspect, embodiments of the present disclosure further provide a text generation apparatus, which includes:

[0010] An encoding execution module, configured to input the obtained original text into a trained text encoding model to obtain text feature information;

[0011] A decoding execution module, configured to generate a target text corresponding to the original text based on the text feature information and in combination with a trained text decoding model;

[0012] Wherein, the text decoding model includes a text prediction layer, the node information of a set number of nodes included in the text prediction layer is determined based on the text feature information, and the target words included in the target text and the combination order of each target word are determined by the node information of each node and the topological structure between the nodes.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, which includes:

[0014] One or more processors;

[0015] A storage device, configured to store one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the text generation method provided in any embodiment of the present disclosure.

[0017] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the text generation method provided in any embodiment of the present disclosure is implemented.

[0018] The technical solution of the embodiment of the present disclosure specifically discloses a text generation method, device, computer device and storage medium. The text generation method includes: inputting the obtained original text into a trained text encoding model to obtain text feature information; based on the text feature information, in combination with a trained text decoding model, generating a target text corresponding to the original text; wherein, the text decoding model includes a text prediction layer, the node information of a set number of nodes included in the text prediction layer is determined based on the text feature information, and the target text is determined by the node information of each node. The above technical solution realizes the parallel determination of the node information of each node in the newly added text prediction layer and the parallel determination of each target word in the generated text, reducing the text generation delay; at the same time, through the node information of each node in the newly added text prediction layer, a one-to-one correspondence between each word in the generated text and the matching node can be realized, thus better avoiding the occurrence of consecutive repeated words in the generated text; in addition, through the topological structure between each node, the combination order of each word in the generated text can be limited, thereby ensuring the relevance of the context in the generated text, and thus improving the generation quality of the generated text and ensuring the text accuracy. Description of the Drawings

[0019] To more clearly illustrate the technical solutions of the exemplary embodiments of the present disclosure, the following briefly introduces the accompanying drawings required for describing the embodiments. Obviously, the accompanying drawings introduced are only the drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic flowchart of a text generation method provided in Embodiment 1 of the present disclosure;

[0021] Figure 1a The application effect diagram of the existing text generation model in the machine translation scenario is given;

[0022] Figure 1b The structural diagram of the text decoding model adopted in the text generation method provided in this embodiment is given;

[0023] Figure 1c The application effect diagram of the text generation model involved in this embodiment in the machine translation scenario is given;

[0024] Figure 2 It is a schematic flowchart of a text generation method provided in the embodiment of the present disclosure;

[0025] Figure 2a The schematic diagram of a part of the network structure in the text decoding model adopted in the text generation method provided in this embodiment is given;

[0026] Figure 2b One of the example diagrams for calculating the node transition matrix in the text generation method provided in this embodiment is given;

[0027] Figure 2c The example diagram of the fully connected structure in the text prediction layer involved in the text generation method provided in this embodiment is given;

[0028] Figure 3 It is a schematic structural diagram of a text generation device provided in Embodiment 3 of the present disclosure;

[0029] Figure 4 It is a schematic structural diagram of an electronic device provided in Embodiment 4 of the present disclosure. Detailed implementation manners

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0031] It should be understood that the various steps recited in the method embodiments of the present disclosure can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0032] As used herein, the term "comprising" and its variations are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence relationship of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "plural" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] Embodiment 1

[0036] Figure 1 It is a flowchart of a text generation method provided for Embodiment 1 of the present disclosure. This embodiment is applicable to the situation of text generation. This method can be executed by a text generation device, which can be implemented by software and / or hardware, and can be configured in a terminal and / or a server to implement the text generation method in the embodiments of the present disclosure.

[0037] It should be noted that in traditional text generation models, sample data consisting of one input text and multiple output texts is usually used for training and learning. After training a conventional text generation model in this training format, in actual applications, there is a problem of mixed output of predicted words in the generated target text. This is mainly because it is impossible to distinguish which output text the predicted word comes from during the training stage, and the possible outputs of the predicted words included in multiple output texts are mixed together, thus unable to guarantee the quality of text generation.

[0038] Exemplarily, Figure 1a the application effect diagram of the existing text generation model in the machine translation scenario is given. As Figure 1a shown, the input text can be the Chinese sentence "I went to the movie theater". In the application scenario of machine translation, the purpose of the existing text generation model 11 is to generate the English text of the above Chinese sentence. When training the existing text generation model 11, there can be multiple English output samples, such as: "I went to the movie theater" and "I just went to the cinema". After the training is completed, when actually performing English machine translation on "I went to the movie theater", it is possible to mix the words in the above output samples and generate an incorrect predicted text of "I went went the the theater".

[0039] A text generation method provided in this embodiment improves the traditional text generation model by adding a text prediction layer. Through each node included in the added text prediction layer, high-quality generated text can be obtained.

[0040] Specifically, as Figure 1 shown, a text generation method provided in Embodiment 1 of this embodiment may include the following steps:

[0041] S101. Input the obtained original text into the trained text encoding model to obtain text feature information.

[0042] It should be known that the text generation method provided in this embodiment is not limited to a certain application scenario. If text generation is required in a certain application scenario, training samples can be collected in that application scenario for training the text generation model. Among them, the text generation model can include two parts in structure, one part is the text encoding model, and the other part is the text decoding model.

[0043] In this embodiment, the original text is equivalent to the input text before text generation, and the content of the original text may be different in different application scenarios. For example, in the machine translation scenario, assuming Chinese-English translation, the original text can be the Chinese text to be translated; if it is English-Chinese translation, the original text can be the English text to be translated.

[0044] In this embodiment, the text encoding model can be used to encode the original text to obtain the text feature information of the original text. Among them, the model structure of this text encoding model can directly reuse the text encoding model in the traditional text generation model, and can be trained and learned through the sample data provided in different application scenarios, so that the output text feature information can meet the text generation requirements of the application scenario. Exemplarily, in the application scenario of machine translation, the output text feature information is mainly used to obtain the translation text corresponding to the original text subsequently.

[0045] In this embodiment, the text feature information is used to characterize the feature information of each word in the originally input original text. This text feature information can be represented by a text feature matrix. Generally, the number of text feature vectors included in this text feature matrix is the same as the number of words included in the original text.

[0046] S102. Based on the text feature information, combined with the trained text decoding model, generate the target text corresponding to the original text.

[0047] In this embodiment, after obtaining the text feature information output by the text encoding model through the above steps, this text feature information can be further used as input data and input into the text decoding model. In this embodiment, the text decoding model includes a text prediction layer. The node information of the set number of nodes included in the text prediction layer is determined by the text feature information, and the target words included in the target text and the combination order of each target word are determined by the node information of each node and the topological structure between the nodes.

[0048] Specifically, compared with the text decoding model in the traditional text generation model, the text decoding model adopted in this step includes a text prediction layer, and the text prediction layer includes a certain number of nodes. Among them, the target text of the original text can be effectively determined through the node information of each node and the topological structure between the nodes. It can be known that the text decoding model in this embodiment is also trained and learned through the sample data provided in different application scenarios, so that the output target text can meet the text generation requirements of the application scenario.

[0049] Continuing with the above description, the text prediction layer contains a set number of nodes. All the nodes can be used to construct the graph required for text generation, and the node information of each node can be determined by the text feature information. In this embodiment, the specific value of the set number is greater than the number of words contained in the original text. It can be used as the size of the graph required for graph construction in the text prediction layer, and can also be used as the possible prediction length of the text to be generated, that is, the number of words contained in the text to be generated will not be greater than this set number. The node information of each node contained in the text prediction layer can be determined by the text feature information. Exemplarily, the text feature information can be combined with some parameter information for fully connected processing, and finally the relevant feature information of each word in the original text is respectively mapped to the nodes as the node information of the nodes.

[0050] In this embodiment, for the generation logic of the target text, it needs to consider the node information of the nodes in the text prediction layer and the topological structure between the nodes. It can be analyzed that the target text is also composed of individual words, and the words in the target text should have some association with the words in the original text. Among them, through the above-mentioned text encoding model in this embodiment, the text feature information representing each word in the original text can be obtained. Then, through the text decoding model in this embodiment, the text feature information can be converted into the node information of each node included in the text prediction layer through basic decoding processing, which is equivalent to establishing an association between each word in the original text and each node in the text prediction layer.

[0051] Specifically, the text decoding model provided in this embodiment can establish a corresponding relationship between each node and the words in the dictionary through the node information of each node in the text prediction layer, so that each node can correspond to a most matching word.

[0052] In addition, the text encoding model provided in this embodiment can also connect the nodes in the text prediction layer according to certain connection conditions to form a topological structure between the nodes. Based on the formed topological structure between the nodes, the connection relationship between the nodes can be clarified. According to the trained learning parameters in the text prediction layer and combined with the topological structure between the nodes, the transition probability from one node to another connected node can be determined. Finally, based on the words corresponding to each node and the transition probability from the node to other connected nodes, the target node can be selected from each node. Since the nodes and words are in one-to-one correspondence, the target words required for generating the target text are also determined accordingly when the target node is selected; in addition, the combination order of the target words in the generated target text can also be determined by the connection relationship between the nodes represented by the topological structure between the nodes. Through the above logic, a target text that avoids consecutive repeated words and has a clear context relationship can be determined relative to the original text.

[0053] Based on this embodiment, the text decoding model is further optimized. Preferably, the text decoding model may specifically include: a position information input layer, a basic decoding sub-model, and a text prediction layer;

[0054] Among them, the position information input layer includes a set number of node position parameters, and the set number is used to determine the number of nodes included in the text prediction layer; the node information of the set number of nodes included in the text prediction layer is determined by each of the node position parameters and the text feature information in combination with the basic decoding sub-model.

[0055] In the above optimization embodiment, the text decoding model includes not only the text prediction layer, but also the position information input layer and the basic decoding sub-model. In terms of structural connection, the information output by the position information input layer is transmitted to the basic decoding sub-model, and the information output by the basic decoding sub-model is respectively transmitted to each node in the text prediction layer.

[0056] In this embodiment, the position information input layer can be specifically understood as an information input layer for predicting the size of the graph required for the directed acyclic graph to be generated in the text prediction layer during text generation. The size of the graph predicted in this position information input layer is actually the number of nodes required to construct the graph, and the value of the graph size can be preferably set to a multiple of the number of words included in the original text. It can be known that this number of nodes determines the number of nodes included in the text prediction layer, that is, the set number representing the number of nodes in the text prediction layer is preset in this position information input layer; when the graph size is set to n, it is equivalent to determining that the number of nodes included in the text prediction layer is n.

[0057] Continuing the above description, in the position information input layer, in addition to presetting the number of nodes included in the text prediction layer, it is also necessary to preset the position information of each node. In this embodiment, node position parameters are used to represent the position information of each node. The node position parameter can be understood as the position parameter given to the nodes required to construct the graph, and each node position parameter represents a corresponding node in the text prediction layer; at the same time, this node position parameter is also equivalent to one of the learning parameters obtained through training in the text decoding model. Through training iteration, the node position parameter can be adjusted accordingly until stable parameter information is obtained after the training ends.

[0058] In the specific implementation of text generation, the text feature information input by the text encoding model and the position parameters of each node can be used as the inputs of the basic decoding sub-model in the text decoding model respectively. The basic decoding sub-model can output vector information with the same number of nodes as those included in the text prediction layer, which are used as the node information of the corresponding nodes respectively. Among them, the basic decoding sub-model can include: a self-attention mechanism self-attention network structure and a cross-attention mechanism cross-attention network structure, which is equivalent to reusing the text decoding model in the traditional text generation model.

[0059] Exemplarily, Figure 1b a structural diagram of the text decoding model adopted in the text generation method provided in this embodiment is given. As Figure 1b shown, the text decoding model 12 includes an input layer. Specifically, the input layer includes two different input branches. One input branch is the position information input layer 121 for inputting the graph size and node position information. The position information input layer 121 includes n determined node position parameters g; the other input branch is used to input the text feature information output by the text encoding model; the text decoding model 12 further includes a basic decoding sub-model 122 and a text prediction layer 123. The basic decoding sub-model 122 can include an m-layer network structure composed of a self-attention mechanism and a cross-attention mechanism; the text prediction layer 123 includes n nodes with the same number as the number of node position parameters; finally, the target text of the original text is output through the output layer 124 of the text decoding model 12.

[0060] A text generation method provided in Embodiment 1 of the present invention realizes the parallel determination of the node information of each node in the newly added text prediction layer and the parallel determination of each target word in the generated text, reducing the text generation delay; at the same time, through the node information of each node in the newly added text prediction layer, it is possible to realize the one-to-one correspondence between each word in the generated text and the matching node, thus better avoiding the appearance of consecutive repeated words in the generated text; in addition, through the topological structure between the nodes of each node, the combination order of each word in the generated text can be limited, thereby ensuring the relevance of the context in the generated text, and thus improving the generation quality of the generated text and ensuring the text accuracy.

[0061] As an alternative embodiment of this embodiment, in this alternative embodiment, the method further optimizes and adds:

[0062] Based on the set loss function generation strategy, the learning parameters of the constructed text decoding model are trained to obtain the trained text decoding model;

[0063] Among them, the learning parameters include: the node position parameters involved in the position information input layer included in the text decoding model, the basic model parameters involved in the included basic decoding sub-model, and the node-related parameters of each node included in the included text prediction layer.

[0064] For a traditional text generation model as Figure 1a shown, another problem existing in its training stage is that in the sample data participating in training, there is an input text and multiple output texts, so there will be a label inconsistency problem in the training stage. Specifically, for the same input text, there are multiple possible different output texts. In the model training stage, when learning the learning parameters at the same position, the predicted words corresponding to them may come from different output texts, thus causing training difficulties.

[0065] Based on this, on the one hand, this embodiment improves the network structure of the text decoding model. For example, a text prediction layer is added, and a value greater than the number of words contained in the text is used as the number of nodes, so that each node can correspond to a word in the output text. On the other hand, sample data improvement and loss function improvement are carried out in the training stage.

[0066] For sample data improvement, this embodiment can use single-sample data, that is, one input text only corresponds to one output text to form a piece of sample data; for loss function improvement, a loss function generation strategy is given, and this strategy mainly considers the nodes in the text prediction layer added in the text decoding model. Exemplarily, this strategy can first consider the possible paths formed between nodes, and consider the generation probability of generating the output text through the formed paths, and then combine the generation probabilities of each path to generate the loss function.

[0067] In this alternative embodiment, through the determined loss function and the improved sample data in the set form, the learning parameters in the created text decoding model can be adjusted by backpropagation, and finally a text decoding model with higher accuracy can be obtained.

[0068] It can be known that the training of the text decoding model is equivalent to the adjustment of each learning parameter in the model. The learning parameters included in this text decoding model can include the node position parameters in the position information input layer; it can also include the respective weight parameters involved in the basic decoding sub-model; it can also include the node-related parameters set for each node in the text prediction layer. The node-related parameters can be used for determining the prediction nodes related to the generated text and the matching of the nodes to the predicted words in the dictionary.

[0069] In this alternative embodiment, the learning parameters of the constructed text decoding model can be further trained based on a set loss function generation strategy, and obtaining the trained text decoding model can be specifically implemented as follows:

[0070] a0. Obtain at least one set of sample data, where one set of sample data includes an original sample text and a corresponding single target sample text.

[0071] In this embodiment, multiple sets of sample data can be obtained to input different sample data in each training iteration. Compared with the existing sample data, in this embodiment, it is preferably that one set of sample data includes an original sample text and a target sample text.

[0072] b0. In the current iteration, after encoding the original sample text in one set of sample data using the text encoding model, input it into the current text decoding model.

[0073] In this embodiment, the current iteration can be understood as possibly the first iteration or a training iteration to be executed in the iteration loop, and the training logic executed in each iteration is the same. The current text decoding model can be understood as the text decoding model to be trained in the current iteration. In this step, the original sample text can be first input into the trained text encoding model for encoding processing, and then input into this current text decoding model.

[0074] c0. Based on the current text decoding model, determine the probability value corresponding to generating the target sample text from the original sample text through each text prediction path.

[0075] In this embodiment, through the network structure included in the current text decoding model and the current parameter values of the learning parameters in the network structure, the original sample text can be processed. Among them, each node in the text prediction layer in the current text decoding model can form various text prediction paths, and a prediction text can be generated through each text prediction path. In this step, the probability value that the prediction text is the target sample text can be determined as the probability value corresponding to generating the target sample text through the text prediction path. This step is equivalent to one of the execution logics in the loss function generation strategy, and the determined probability values are specifically used to determine the loss function value adopted in the current iteration.

[0076] Among them, each text prediction path is formed based on the nodes in the text prediction layer in combination with a set algorithm. Exemplarily, during the execution of this step, all the paths formed by the connections between the nodes can be used as text prediction paths respectively. If all the paths are directly selected as text prediction paths, a large amount of computing resources will be occupied during the path calculation to implement model training. In this embodiment, it is considered to use the dynamic programming algorithm in the calculation of all paths to avoid repeated operations of the same logic, thereby saving computing resources and improving the training time. At the same time, this embodiment can also consider using a certain algorithm to select a part of the paths from all the paths formed by the node connections as text prediction paths.

[0077] d0. Based on each of the probability values and the loss function generation formula, determine the current loss function value, and based on the current loss function value, adjust the learning parameters in the current text decoding model through backpropagation to obtain a text decoding model for the next iteration.

[0078] In this embodiment, the above-determined probability values can be substituted into the pre-set loss function generation formula to determine the current loss function value in the current iteration. Among them, the loss function generation formula is expressed as taking the logarithm of the sum of each of the probability values and taking the negative of the logarithmic operation result.

[0079] e0. Take the next iteration as the new current iteration, and return to continue executing step b0 until the iteration end condition is met to obtain a trained text decoding model.

[0080] In this alternative embodiment, the iteration receiving condition can be that the current loss function value determined in the iteration logic is within a set threshold range, or the number of iterations reaches a set number threshold.

[0081] Through the model training logic given in this alternative embodiment, the problem of inconsistent labels in the training samples that occurs during the model training stage can be better avoided, so that each node in the text encoding model can correspond one by one to the words that appear in the text to be generated.

[0082] Exemplarily, Figure 1c The application effect diagram of the text generation model involved in this embodiment in the machine translation scenario is given. As Figure 1cAs shown, the input text can also be the Chinese "I went to the cinema". In the application scenario of machine translation, the text generation model 13 (the model includes a text prediction layer) used in this embodiment can generate the English text of the above Chinese sentence. When training the text generation model 13 used in this embodiment, the English text samples used may have only "I went to the movie theater" or only "I just went to the cinema". After the training is completed, each word in the English text sample corresponds to a processing node in the text generation model 13 (in Figure 1c In the example, a processing node can be represented by predicting each word presented in the text); this is equivalent to the text generation model 13 adopted in this embodiment being able to determine the most matching word for each processing node, and based on the connection relationship between the processing nodes, a combined path that best matches the context relationship can be determined from the connection paths formed by the processing nodes.

[0083] When the text generation model trained in this embodiment is used to perform machine translation of "I went to the movie theater" in English, only the words corresponding to the processing nodes in the combination path can be selected for combination, thereby forming an output target text. For example, based on one of the determined combination paths, the corresponding output text can be expressed as: "I went to the movie theater". Compared with Figure 1a The erroneous text output in the example is “I went went the the theater”. The text output in this embodiment avoids the duplication of word connections and ensures the coherence of the context.

[0084] Embodiment 2

[0085] Figure 2 A flow chart of a text generation method provided by an embodiment of the present disclosure is given. This embodiment is a further optimization of the above embodiment. In this embodiment, the target text corresponding to the original text is further generated based on the text feature information in combination with the trained text decoding model as follows: the text feature information and the position parameters of each node in the position information input layer are input into the basic decoding sub-model; the set number of initial text prediction vectors output by the basic decoding sub-model are obtained, and each of the initial text prediction vectors is used as the node information of each node in the text prediction layer; a directed acyclic graph is constructed based on each of the nodes, the topological structure between the nodes is determined, and the target text of the original text is determined in combination with the information of each node.

[0086] like Figure 2 As shown, a text generation method provided in this embodiment 2 specifically includes the following steps:

[0087] S201. Input the obtained original text into the trained text encoding model to obtain text feature information.

[0088] Exemplarily, the text feature information may be a feature matrix containing the feature vectors corresponding to each word in the original text.

[0089] S202. Input the text feature information and the position information into each node position parameter in the layer, and input them into the basic decoding sub-model.

[0090] In this embodiment, the text decoding model includes a position information input layer, and the position information input layer includes the position information (node position parameters) of the nodes in the graph to be constructed in the text decoding model and the graph size of the graph to be constructed (mainly characterized by the number of node position parameters included).

[0091] This step can use both the text feature information and each node position parameter as input information and input them into the basic decoding sub-model in the text decoding model.

[0092] Figure 2a The schematic diagram of part of the network structure in the text decoding model used in the text generation method provided in this embodiment is given. As Figure 2a shown, the position information input layer in the text decoding model and the basic decoding sub-model 20 are given; among them, the position information input layer includes 9 (graph size) node position parameters 21. Each node position parameter 21 and the text feature information 22 output by the text encoding model can be input into the basic decoding sub-model 20.

[0093] S203. Obtain the set number of initial text prediction vectors output by the basic decoding sub-model, and use each of the initial text prediction vectors as the node information of each node in the text prediction layer.

[0094] This step can obtain the processing information output by the basic decoding sub-model. The processing information may specifically include a set number of initial text prediction vectors. The set number is the same as the number of node position parameters in the position information input layer. This step can also correspond the obtained initial text prediction vectors to each node in the text prediction layer and use them as the node information of the nodes.

[0095] Continuing from the above Figure 2a , it can be seen that Figure 2a also gives the node set in the text prediction layer, and the node set also includes 9 nodes 23; each initial text prediction vector output by the basic decoding sub-model 20 can be in one-to-one correspondence with the nodes 23 to be used as the node information of each node 23.

[0096] S204. Construct a directed acyclic graph based on each of the nodes, determine the topological structure between the nodes, and determine the target text of the original text in combination with the information of each node.

[0097] The above steps are equivalent to assigning node information to each node in the text prediction layer, so that each node in the text prediction layer is associated with the actual original text.

[0098] This step is equivalent to taking the text prediction layer as the execution subject, which mainly performs subsequent processing of text generation based on the node information of each node, so as to generate the target text of the original text.

[0099] The analysis of the execution logic of this step can be described as follows: After each node is assigned node information, they are still individual nodes, and there is no association between the nodes; considering that there is a context association between the words in the text to be generated, and each word is related to the nodes in the text prediction layer, therefore, this step needs to establish the association between the nodes, and the association between the nodes can be achieved by constructing a graph. Considering that the text to be generated is directed and acyclic, this step can construct a directed acyclic graph based on each node.

[0100] Following the above analysis, there needs to be a context association between the words in the text to be generated. After determining that a node can represent a word, to determine the context association between the words, it can be converted into the association between the nodes, and the association between the nodes can be reflected by the weight of the edge formed after the nodes are connected in the directed acyclic graph. In this embodiment, it is considered to represent the weight of the edge formed by two nodes through the transition probability from one node to another node. After determining the transition probability between the nodes, the higher the transition probability between the two nodes, the greater the association between the two nodes can be considered.

[0101] Based on the above analysis, the execution logic of generating the target text corresponding to the original text based on the node information in this step can be described as follows: 1) Establish a directed connection between each node to form a directed acyclic graph, and determine the transition probability from the source node to the target node among the two connected nodes, where the source node is the out-end node in the directed connection between the two nodes, and the target node is the in-end node in the directed connection between the two nodes; 2) Determine the predicted word corresponding to each node; 3) According to the transition probability between the nodes and the predicted words corresponding to the nodes, select the target words, and finally combine the target words in the obtained combination order to form the target text.

[0102] Further, this embodiment gives one implementation manner of constructing a directed acyclic graph based on each of the nodes, determining the topological structure between the nodes, and determining the target text of the original text in combination with the information of each node. The implementation steps include the following steps a1 to c1, specifically:

[0103] a1. Predict the node labels of each node in the text prediction layer, construct a directed acyclic graph, and obtain the topological structure between nodes.

[0104] Exemplarily, the construction of the directed acyclic graph is used to determine the connection relationship between nodes. Considering the directivity of the constructed graph, in this embodiment, directed connections are made based on the node labels of the nodes. For example, assuming there are 9 nodes, based on the ascending order of the node labels, node v1 will establish directed connections with nodes v2 to v9 respectively, while node v2 can only establish directed connections with v3 to v9, and so on. The last node v9 will no longer have a directed connection. After determining the directed acyclic graph, it is equivalent to determining the topological structure between nodes.

[0105] b1. Determine the node transition matrix corresponding to the text prediction layer according to the topological structure between nodes and the node information of each node.

[0106] In this embodiment, the topological structure between nodes includes the connection relationship between nodes. Based on the connection relationship between nodes, it can be known which nodes each node is connected to, and the existing connections are directed connections. In this embodiment, the row and column values of the node transition matrix are respectively the number of nodes included in the text prediction layer. And considering the directivity of node connections, the node transition matrix can preferably be an upper triangular matrix. For a valid element value in the node transition matrix, it represents that there is a directed connection between the node corresponding to the corresponding row and the node corresponding to the corresponding column, and mainly the transition probability between the two nodes determined through the corresponding calculation logic.

[0107] In this embodiment, for the determination of the transition probability between nodes, one implementation logic can be described as follows: for two connected nodes, their node information can be obtained, where the node information can be represented by a feature vector. Then, the feature vectors representing the node information of the two nodes can be multiplied, and the obtained product vector can be normalized to be used as the transition probability between the two nodes.

[0108] For the determination of the transition probability between nodes, another implementation logic can also be described as follows: first, obtain the node-related parameters set for each node in the text prediction layer, such as the first learning parameter and the second learning parameter, which are mainly used for the determination of the transition probability; among them, the node-related parameters of each node exist in the text decoding model, and after the text decoding model is trained, they can have fixed parameter values; then, for two connected nodes, the transition probability can be further determined according to the product vector obtained by multiplying the node information by the node-related parameters.

[0109] Among them, for the above implementation of determining the transition probability between two nodes by combining node information with node-related parameters, the following exemplary description is given: for node v iand node v j For example, node v i is connected to node v j , node v i and node v j The calculation of the transition probability can be described as follows: Determine the product of the initial text prediction vector (node information) of node v i and the first learning parameter (denoted as the first product); Determine the product of the initial text prediction vector (node information) of node v j and the second learning parameter (denoted as the second product); Normalize the product result of the first product and the second product, and the normalized result can be regarded as the transition probability between node v i and node v j .

[0110] Based on the above description, it can be known that after determining the transition probability between two connected nodes, based on each transition probability, the node transition matrix of the text prediction layer can be formed.

[0111] Furthermore, in this embodiment, determining the node transition matrix corresponding to the text prediction layer according to the node topology structure and the node information of each node can be specifically implemented as follows:

[0112] b11. For each node, determine the adjacent nodes that the node is directed-connected to from the node topology structure.

[0113] Among them, through the constructed directed acyclic graph, after obtaining the node topology structure, it is easy to determine other nodes that have a directed connection with the node, and these nodes can be regarded as the adjacent nodes of the node.

[0114] b12. According to the node information of the node and each adjacent node, determine the transition probability from the node to each adjacent node.

[0115] Exemplarily, in one implementation manner, the calculation of the transition probability p vi->vj from node vi to node vj can be described as follows: where softmax represents normalization, represents the scale size of this text prediction layer (d is determined in the construction stage), and Vi and Vj respectively represent the node information vectors of node vi and node vj.

[0116] In another exemplary implementation, the implementation logic can be summarized as follows: for each node, according to the node information of the node and its corresponding adjacent nodes, the first learning parameter, and the second learning parameter, combined with the probability transition formula, determine the transition probability from the node to each adjacent node, where the first learning parameter and the second learning parameter are both node-related parameters corresponding to the node. Referring to the above description, the probability transition formula can be expressed as:

[0117]

[0118] The node information vectors of point vi and node vj; in addition, in this formula, W1 represents the first learning parameter related to the node; W2 represents the second learning parameter related to the node; p vi->vj represents the transition probability from node vi to node vj.

[0119] b13. Based on each of the transition probabilities, form a node transition matrix corresponding to the text prediction layer.

[0120] It can be known that based on the above steps b12 and b13, the transition probabilities between each node and its adjacent nodes can be calculated, and based on each transition probability, a node transition matrix can be formed.

[0121] Exemplarily, Figure 2b Figure 2b shows one of the example diagrams for calculating the node transition matrix in the text generation method provided in this embodiment. As Figure 2b shown, the transition probabilities for each node included in the text prediction layer are calculated. Figure 2b In Figure 2b, E represents the calculated node transition matrix. It should be noted that Figure 2b shows partial connections of each node and the corresponding transition probabilities of the connections. For example, the transition probability from v1 to v2 is 0.3; the transition probability from v1 to v3 is 0.7, etc. It can be known that in the node transition matrix E, the sum of the transition probabilities of each row is 1.

[0122] c1. According to the node information of each of the nodes and the node transition matrix, determine the target text of the original text.

[0123] In this embodiment, after determining the node transition matrix, it is equivalent to determining the weights of each edge formed by the connections in the directed acyclic graph. In this embodiment, a prediction path can be selected through a prediction path selection strategy; exemplarily, for the selection of the prediction path, one implementation can be described as follows: along the direction of the node connection line, on the premise that the out-end node is fixed, select the in-end node with the highest transition probability from the out-end node, and use the edge between the two nodes as one of the edges in the prediction path; then repeat the above logic when selecting a new out-end node, and finally select all the edges in the prediction path, and thus also determine each target point constituting the prediction path.

[0124] As Figure 2b shown, through the above logical description, the predicted path can be determined as v1->v3->v4->v5->v6->v9; the target points included are A = {v1, v3, v4, v5, v6, v9}.

[0125] Meanwhile, in this step, according to the information of each node and combined with the fully connected layer existing in the text prediction layer, the probability information of each node and each word included in the dictionary can be determined. Here, the dictionary can be the pre-created vocabulary information, which contains various words required for text generation, and each word can be represented in the form of a vector. Based on the fully connected layer existing in the text prediction layer, the nodes in the previous layer of the fully connected layer can be the nodes in the graph of this embodiment, and the nodes in the next layer can be the word nodes in the dictionary. The fully connected process can be to calculate the matching probability from each node in the graph to each word node in the dictionary, and the calculation form can be realized through the full connection of the node information of the node and the word vector of the word node.

[0126] After determining the predicted path and the matching vector from the node to the word as above, the target word corresponding to each node in the predicted path can be determined, and finally the target text is formed based on the combination of each target word. It should be noted that in this embodiment, the execution order of determining the predicted path and the matching probability is not determined, and it can also determine the predicted path after determining the matching probability. As long as the generation of the target text can be completed.

[0127] Based on the above embodiment, this embodiment can also specifically describe the above step c1 "determine the target text of the original text according to the node information of each of the nodes and the node transition matrix".

[0128] Exemplarily, after obtaining the node transition matrix corresponding to the text prediction layer and the node information of each node, it can be realized through the execution logic of steps c11~c13 provided in this embodiment.

[0129] It should be noted that in addition to the nodes required for constructing the directed acyclic graph, the text prediction layer also includes a fully connected structure. The fully connected structure can regard the node information of each node in the directed acyclic graph as input information, and the next layer in the fully connected structure can be regarded as the word nodes formed by each word in the dictionary. In the fully connected structure, the nodes in the graph and the word nodes in the dictionary can be connected through connection lines. The connection weight of each connection line in the fully connected structure can be the third learning parameter determined for the connection between each node and the word after training the text decoding model.

[0130] Figure 2c Fig. shows an example diagram of the fully connected structure in the text prediction layer involved in the text generation method provided in this embodiment. As Figure 2cAs shown, above each node shown in the directed acyclic graph, there is a fully connected structure 24 for determining the prediction words associated with the node. It should be noted that Figure 2c also includes a result output layer. On the result output layer, only the prediction words matching the nodes in the directed acyclic graph are shown. For example, the word matching node v1 is "I"; the word matching node v2 is "just"; the word matching node v3 is "went", etc.

[0131] c11. According to the node information of each of the said nodes, through the fully connected layer in the text prediction layer, determine the matching probability of each of the said nodes to each word in the preset vocabulary.

[0132] The specific implementation of this step, the execution logic can be described as that each node has a connection with each word in the dictionary. In this step, both the node and the word can also represent the corresponding information through vectors. Thus, for the matching probability between the node and the word, if the connection weights in the fully connected structure are re-determined during the training stage of the text decoding model, first obtain the third learning parameter obtained from training, and then determine the vector product of the corresponding third learning parameter and the vector of the corresponding node information and word information; if the connection weights are not re-determined during the training stage of the text decoding model, but directly share the word features used by the text encoding model, directly determine the vector product of the corresponding node information and word information; then the vector product of the node relative to all words can also be determined, and after normalization, it can be used as the matching probability of the node to the word.

[0133] Among them, the fully connected layer is constructed within the text prediction layer, which includes a fully connected structure for processing the matching probability, and the fully connected structure can perform a fully connected process relative to each node.

[0134] c12. According to the node transition matrix and the matching probability of each node to each word, determine the predicted node and the corresponding target word.

[0135] In this embodiment, the predicted node can be considered as the key node on which the generation of the target text depends among the nodes in the text prediction layer. Based on the matching probability corresponding to each predicted node, the prediction word matching the predicted node can be determined, and the prediction word can be regarded as the target word included in the target text.

[0136] In this embodiment, prediction points can be obtained by determining a prediction path based on a node transition matrix in a text prediction layer, and then the target word of the prediction point can be determined by the matching probability from nodes to words; alternatively, prediction nodes and target words can be determined based on the node transition matrix and the matching probability from nodes to words, and further a prediction path can be determined based on each prediction node to be used to combine target words to form a target text; or first, the prediction words corresponding to each node can be determined based on the matching probability from nodes to words, and then a prediction path can be determined in a directed acyclic graph through a search algorithm, and finally the target word required for text generation can be selected.

[0137] c13. Combine the target words to form a target text of the original text.

[0138] In this step, the determined target words are combined according to the connection direction between the corresponding nodes in the text prediction layer. Among them, only one combination order can be determined for each target word, and finally the final target text can be obtained according to this combination order. This target text is equivalent to the result obtained after performing text generation processing on the original text.

[0139] On the basis of the above optimization, this embodiment gives a further optimization of the above step c13. Exemplarily, for determining prediction nodes and corresponding target words according to the node transition matrix and the matching probability from each node to each word, this embodiment provides a preferred implementation manner, which can be specifically described as:

[0140] Determine at least one prediction node according to the maximum transition probability corresponding to each node in the node transition matrix.

[0141] Among them, the determination of the maximum transition probability of each node has an order. It first starts from the node corresponding to the starting node label, and this node can be used as the first prediction node. Among the transition probabilities corresponding to the connections between this prediction node and each adjacent node, the maximum transition probability of this prediction node can be determined, and the adjacent node corresponding to this maximum transition probability can be regarded as a new prediction node; then, the maximum transition probability can be determined again for the new prediction node, and a new prediction node can be determined therefrom; through the above logic, prediction nodes can be determined cyclically until the last node is reached, and the last node can also be used as the last prediction node. Thus, at least one prediction node can be obtained in this step (in the case of one, the starting node is also the ending node).

[0142] For each prediction node, determine the maximum matching probability from the matching probabilities from the prediction node to each word, and determine the word corresponding to this maximum matching probability as the target word.

[0143] Among them, for each of the above-determined prediction nodes, after knowing the matching probabilities between each prediction node and each word, the maximum matching probability can also be determined from these matching probabilities. Furthermore, the prediction word corresponding to this maximum matching probability can be obtained, and this prediction word is equivalent to the target word corresponding to this prediction node. It can be known that through the determination order of the prediction nodes, a combination path for the target word combination can be determined, and this combination path can be used as the final target text generation.

[0144] Exemplarily, for the further optimization of the above step c13, this embodiment also provides another preferred implementation manner. It should be noted that, different from the above implementation logic, the implementation logic of this method lies in simultaneously considering the influence of the transition probability in the node transition matrix and the matching probability between the node and the word on the prediction node. It can multiply the transition probability and the matching probability, and determine the prediction node based on the product result.

[0145] Among them, the specific steps of this implementation manner can be described as follows:

[0146] 1) Use the node corresponding to the starting node label as the current node.

[0147] Among them, this current node can be recorded as the first prediction node.

[0148] 2) Obtain the current transition probabilities from the current node to each adjacent node from the node transition matrix.

[0149] 3) Determine the product values of each of the current transition probabilities and the matching probabilities between the current node and each word.

[0150] 4) Select the maximum product value from each of the product values, and use the adjacent node and word associated with the maximum product value as the prediction node and the target word respectively, and add the prediction node and the target word to the cache table in an associated manner.

[0151] Among them, the matching probability and the transition probability corresponding to the maximum product value can be known. With the current node as a reference, it can be known that the above matching probability is relative to the word corresponding to the current node, and this word is recorded as a target word. It can also be known that the above transition probability is relative to the adjacent node corresponding to the current node, and this adjacent node can be recorded as another prediction node.

[0152] 5) Use the prediction node as the new current node, and re-execute the selection operation for the current adjacent node corresponding to the current node until the loop end condition is reached.

[0153] It can be seen that in this execution logic, the loop processing is also carried out in the order of the directed connection of the nodes. Thus, each prediction node and target word that meet the conditions can be determined.

[0154] Similarly, in the process of determining the prediction nodes as described above, it is equivalent to determining the combination order of the target word combinations adopted.

[0155] Exemplarily, for the further optimization of the above step c13, this embodiment also provides another preferred implementation manner. Different from the above two implementation manners, this implementation manner mainly considers the situation where different nodes may correspond to the same word. This embodiment's manner is equivalent to proposing a way to determine the target word based on this situation.

[0156] Among them, the specific steps of this implementation manner can be described as:

[0157] 1) Based on the matching probabilities from each node to each word, determine the corresponding maximum matching probabilities, and determine the words corresponding to each of the maximum matching probabilities as the prediction words of the corresponding nodes.

[0158] First, through this step, determine the corresponding prediction words for each node in the text prediction layer. Among them, the determination of the prediction words is also implemented using the logic of the maximum matching probability.

[0159] 2) According to the pre-set path search algorithm, combine the node transition matrix and the prediction words of each of the nodes to determine the prediction path with the highest weight.

[0160] The main purpose of this step is to determine each candidate text generation path for each node in the text prediction layer based on the node label order, and based on the node transition matrix, determine the transition probability of the edge between two nodes in each candidate text generation path; then, through the path search algorithm combined with the prediction words, determine each candidate prediction path where different nodes represent the same prediction word from each candidate text generation path; and obtain the prediction path with the highest weight from the candidate prediction paths.

[0161] 3) Determine the prediction words corresponding to each prediction node in the prediction path as the corresponding target words.

[0162] For the above three implementation manners of determining the prediction nodes and target words, the first one has the fastest execution speed, but the generation quality of the generated text is relatively low; the second one is in a moderate state in terms of execution speed and text generation quality; the third one has a relatively slow execution speed, but the generation quality of the generated text is relatively high. This embodiment can adopt the above several manners but is not limited to the above manners. In the application scenario, the appropriate implementation manner of the prediction nodes and target words can be considered according to the actual situation to generate the target text.

[0163] A text generation method provided in the second embodiment specifies the implementation process of a text decoding model for generating a target text. By adding a text prediction layer, it is considered to effectively determine target words and prediction nodes in the form of a directed acyclic graph using graph nodes, ensuring the relevance of the context and avoiding the continuous occurrence of repeated words in the generated text. Compared with the prior art, the generation quality of the generated text is improved, and the text accuracy is guaranteed.

[0164] Embodiment Three

[0165] Figure 3 FIG. 3 is a schematic structural diagram of a text generation device provided in Embodiment Three of the present disclosure. This embodiment is applicable to the situation of text generation. The device can be implemented by software and / or hardware, and can be configured in a terminal and / or a server to implement the text generation method in the embodiments of the present disclosure. The device specifically includes: an encoding execution module 31 and a decoding execution module 32.

[0166] Among them, the encoding execution module 31 is configured to input the obtained original text into the trained text encoding model to obtain text feature information;

[0167] The decoding execution module 32 is configured to generate a target text corresponding to the original text based on the text feature information in combination with the trained text decoding model;

[0168] Among them, the text decoding model includes a text prediction layer. The node information of a set number of nodes included in the text prediction layer is determined based on the text feature information, and the target words included in the target text and the combination order of each target word are determined by the node information of each node and the topological structure between the nodes.

[0169] The text generation device provided in Embodiment Three realizes the parallel determination of the node information of each node in the added text prediction layer and the parallel determination of each target word in the generated text, reducing the text generation delay; at the same time, through the node information of each node in the added text prediction layer, a one-to-one correspondence between each word in the generated text and the matching node can be achieved, thus better avoiding the occurrence of consecutive repeated words in the generated text; in addition, through the topological structure between the nodes of each node, the combination order of each word in the generated text can be limited, thereby ensuring the relevance of the context in the generated text, and thus improving the generation quality of the generated text and ensuring the text accuracy.

[0170] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the text decoding model includes: a position information input layer, a basic decoding sub-model, and a text prediction layer;

[0171] The position information input layer includes a set number of node position parameters, and the set number is used to determine the number of nodes included in the text prediction layer;

[0172] The node information of the set number of nodes included in the text prediction layer is determined by combining each of the node position parameters and the text feature information with the basic decoding sub-model.

[0173] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the decoding execution module 32 includes:

[0174] An information input unit, configured to input the text feature information and each of the node position parameters in the position information input layer into the basic decoding sub-model;

[0175] An initial vector output unit, configured to obtain the set number of initial text prediction vectors output by the basic decoding sub-model, and use each of the initial text prediction vectors as the node information of each node in the text prediction layer;

[0176] A text generation unit, configured to construct a directed acyclic graph based on each of the nodes, determine the topological structure between the nodes, and determine the target text of the original text in combination with the information of each node.

[0177] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the text generation unit specifically includes:

[0178] A first execution unit, configured to construct a directed acyclic graph according to the node labels of each node in the text prediction layer, and obtain the topological structure between the nodes;

[0179] A second execution unit, configured to determine the node transition matrix corresponding to the text prediction layer according to the topological structure between the nodes and the node information of each of the nodes;

[0180] A third execution unit, configured to determine the target text of the original text according to the node information of each of the nodes and the node transition matrix.

[0181] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the second execution unit is specifically configured to:

[0182] For each node, determine the adjacent nodes to which the node is directed-connected from the topological structure between the nodes;

[0183] Determine the transition probability from the node to each adjacent node according to the node and the node information of each adjacent node;

[0184] Form the node transition matrix corresponding to the text prediction layer based on each of the transition probabilities.

[0185] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the third execution unit is specifically configured to:

[0186] According to the node information of each of the nodes, determine the matching probabilities of each of the nodes to each word in a preset vocabulary through the fully connected layer in the text prediction layer;

[0187] According to the node transition matrix and the matching probabilities of each node to each word, determine a predicted node and a corresponding target word;

[0188] Based on each of the target words, combine to form a target text of the original text.

[0189] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the specific steps for the third execution unit to execute determining a predicted node and a corresponding target word according to the node transition matrix and the matching probabilities of each node to each word may be:

[0190] According to the maximum transition probabilities corresponding to each node in the node transition matrix, determine at least one predicted node;

[0191] For each predicted node, determine the maximum matching probability from the matching probabilities of the predicted node to each word, and determine the word corresponding to the maximum matching probability as the target word.

[0192] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the specific steps for the fourth execution unit to execute determining a predicted node and a corresponding target word according to the node transition matrix and the matching probabilities of each node to each word may also be:

[0193] Take the node corresponding to the starting node label as the current node;

[0194] Obtain the current transition probabilities of the current node to each adjacent node from the node transition matrix;

[0195] Determine the product values of each of the current transition probabilities and the matching probabilities corresponding to the current node and each word respectively;

[0196] Select the maximum product value from each of the product values, and take the adjacent node and the word associated with the maximum product value as the predicted node and the target word respectively, and add the predicted node and the target word to the cache table in an associated manner;

[0197] Take the predicted node as the new current node, and re - execute the selection operation for the current adjacent node corresponding to the current node until the loop end condition is reached.

[0198] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the specific steps for the fourth execution unit to determine the predicted node and the corresponding target word according to the node transition matrix and the matching probabilities of each node to each word may also be:

[0199] Based on the matching probabilities of each node to each word, determine the corresponding maximum matching probabilities, and determine the words corresponding to each of the maximum matching probabilities as the predicted words of the corresponding nodes;

[0200] According to a preset path search algorithm, combine the node transition matrix and the predicted words of each node to determine the predicted path with the highest weight;

[0201] Determine the predicted words corresponding to each predicted node in the predicted path as the corresponding target words

[0202] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the device may further include: a model training module, configured to train the learning parameters of the constructed text decoding model based on a set loss function generation strategy to obtain a trained text decoding model;

[0203] Wherein, the learning parameters include: the node position parameters involved in the position information input layer included in the text decoding model, the basic model parameters involved in the basic decoding sub-model included, and the node-related parameters of each node included in the text prediction layer included.

[0204] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the model training module may specifically be configured to:

[0205] Obtain at least one set of sample data, where one set of sample data includes an original sample text and a corresponding single target sample text;

[0206] In the current iteration, after encoding the original sample text in a set of sample data using a text encoding model, input it into the current text decoding model;

[0207] Based on the current text decoding model, determine the probability values corresponding to generating the target sample text from the original sample text through each text prediction path, where each text prediction path is formed based on the nodes in the text prediction layer and a set algorithm;

[0208] Based on each of the probability values and the loss function generation formula, determine the current loss function value, and adjust the learning parameters in the current text decoding model through backpropagation based on the current loss function value to obtain a text decoding model for the next iteration;

[0209] Take the next iteration as the new current iteration and continue the training of the learning parameters until the iteration end condition is met, and obtain the trained text decoding model.

[0210] Further, the formula for generating the loss function is expressed as: taking the logarithm of the sum of the probability values and taking the negative of the result of the logarithm operation.

[0211] The above device can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0212] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present disclosure.

[0213] Embodiment 4

[0214] Figure 4 It is a schematic structural diagram of an electronic device provided by Embodiment 4 of the present disclosure. Referring to FIG. 4 below, it shows a schematic structural diagram of an electronic device (such as Figure 4 the terminal device or server) 40 in the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), vehicle terminals (such as vehicle navigation terminals), etc. and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0215] As Figure 4 shown, the electronic device 40 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 41, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 42 or the program loaded from the storage device 48 into the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 are also stored. The processing device 41, the ROM 42, and the RAM 43 are connected to each other through a bus 45. The editing / output (I / O) interface 44 is also connected to the bus 45.

[0216] Typically, the following devices can be connected to the I / O interface 44: an input device 46 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 47 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 48 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 49. The communication device 49 can allow the electronic device 40 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 4 the electronic device 40 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices can be alternatively implemented or had.

[0217] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 49, or installed from the storage device 48, or installed from the ROM 42. When the computer program is executed by the processing device 41, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0218] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0219] The electronic device provided by the embodiment of the present disclosure and the text generation method provided by the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0220] Embodiment Five

[0221] An embodiment of the present disclosure provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, the text generation method provided by the above embodiment is implemented.

[0222] It should be noted that the computer-readable medium described above in the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0223] In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0224] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0225] The above computer-readable medium can be included in the above electronic device; or it can exist separately without being assembled into the electronic device.

[0226] The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to:

[0227] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof. The programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0228] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0229] The units involved in the embodiments described in the present disclosure may be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation to the unit itself in some cases. For example, the first acquisition unit may also be described as "the unit for acquiring at least two Internet protocol addresses".

[0230] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, by way of non-limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0231] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0232] According to one or more embodiments of the present disclosure, [Example 1] provides a text generation method, the method comprising: inputting the acquired original text into a trained text encoding model to obtain text feature information; generating a target text corresponding to the original text based on the text feature information in combination with a trained text decoding model; wherein, the text decoding model includes a text prediction layer, the node information of a set number of nodes included in the text prediction layer is determined by the text feature information, and the target words included in the target text and the combination order of each target word are determined by the node information of each of the nodes and the topological structure between the nodes.

[0233] According to one or more embodiments of the present disclosure, [Example 2] provides a text generation method, which preferably includes: a position information input layer, a basic decoding sub-model, and a text prediction layer in the text decoding model; the position information input layer includes a set number of node position parameters, and the set number is used to determine the number of nodes included in the text prediction layer; the node information of a set number of nodes included in the text prediction layer is determined by combining each of the node position parameters and the text feature information with the basic decoding sub-model.

[0234] According to one or more embodiments of the present disclosure, [Example Three] provides a text generation method. The steps in this method: Based on the text feature information, in combination with the trained text decoding model, generate the target text corresponding to the original text, which preferably includes: input the text feature information and the position information into the position parameters of each node in the layer, and input them into the basic decoding sub-model; obtain the set number of initial text prediction vectors output by the basic decoding sub-model, and use each of the initial text prediction vectors as the node information of each node in the text prediction layer; based on each of the nodes, construct a directed acyclic graph, determine the topological structure between the nodes, and in combination with the node information of each node, determine the target text of the original text.

[0235] According to one or more embodiments of the present disclosure, [Example Four] provides a text generation method. The steps in this method: According to the node labels of each node in the text prediction layer, construct a directed acyclic graph to obtain the topological structure between the nodes; according to the topological structure between the nodes and the node information of each node, determine the node transition matrix corresponding to the text prediction layer; according to the node information of each node and the node transition matrix, determine the target text of the original text.

[0236] According to one or more embodiments of the present disclosure, [Example Five] provides a text generation method. The steps in this method: According to the topological structure between the nodes and the node information of each node, determine the node transition matrix corresponding to the text prediction layer, which preferably includes: for each node, determine the adjacent nodes that the node is directed to connect from the topological structure between the nodes; according to the node and the node information of each adjacent node, determine the transition probability from the node to each adjacent node; based on each of the transition probabilities, form the node transition matrix corresponding to the text prediction layer.

[0237] According to one or more embodiments of the present disclosure, [Example Six] provides a text generation method. The steps in this method: According to the node information of each node and the node transition matrix, determine the target text of the original text, which preferably includes: according to the node information of each node, through the fully connected layer in the text prediction layer, determine the matching probability of each node to each word in the preset vocabulary; according to the node transition matrix and the matching probability of each node to each word, determine the predicted node and the corresponding target word; based on each of the target words, combine to form the target text of the original text.

[0238] According to one or more embodiments of the present disclosure, [Example Seven] provides a text generation method. The steps in this method: determining a prediction node and a corresponding target word according to the node transition matrix and the matching probabilities from each node to each word can be specifically optimized as follows: determining at least one prediction node according to the maximum transition probability corresponding to each node in the node transition matrix; for each prediction node, determining the maximum matching probability from the matching probabilities from the prediction node to each word, and determining the word corresponding to this maximum matching probability as the target word.

[0239] According to one or more embodiments of the present disclosure, [Example Eight] provides a text generation method. The steps in this method: determining a prediction node and a corresponding target word according to the node transition matrix and the matching probabilities from each node to each word can be specifically optimized as follows: taking the node corresponding to the starting node label as the current node; obtaining the current transition probabilities from the current node to each adjacent node from the node transition matrix; determining the product values of each of the current transition probabilities and the matching probabilities corresponding to the current node and each word; selecting the maximum product value from each of the product values, and taking the adjacent node and word associated with the maximum product value as the prediction node and the target word respectively, and associatively adding the prediction node and the target word to the cache table; taking the prediction node as the new current node, and re-executing the selection operation for the current adjacent node corresponding to the current node until the loop end condition is reached.

[0240] According to one or more embodiments of the present disclosure, [Example Nine] provides a text generation method. The steps in this method: determining a prediction node and a corresponding target word according to the node transition matrix and the matching probabilities from each node to each word can be specifically optimized as follows: based on the matching probabilities from each node to each word, determining the corresponding maximum matching probability, and determining the word corresponding to each maximum matching probability as the prediction word for the corresponding node; according to a preset path search algorithm, combining the node transition matrix and the prediction words of each node, determining the prediction path with the highest weight; determining the prediction words corresponding to each prediction node in the prediction path as the corresponding target words.

[0241] According to one or more embodiments of the present disclosure, [Example Ten] provides a text generation method, which further optimizes to include: training the learning parameters of the constructed text decoding model based on a set loss function generation strategy to obtain the trained text decoding model; wherein, the learning parameters include: the node position parameters involved in the position information input layer included in the text decoding model, the basic model parameters involved in the basic decoding sub-model included, and the node-related parameters of each node included in the text prediction layer included.

[0242] According to one or more embodiments of the present disclosure, [Example XI] provides a text generation method. The steps in this method: Based on a set loss function generation strategy, training the learning parameters of the constructed text decoding model to obtain a trained text decoding model can be optimized as follows: Obtain at least one set of sample data, where one set of sample data includes an original sample text and a corresponding single target sample text; In the current iteration, after encoding the original sample text in one set of sample data using the text encoding model, input it into the current text decoding model; Based on the current text decoding model, determine the probability values corresponding to generating the target sample text from the original sample text through each text prediction path, where each text prediction path is formed by combining nodes in the text prediction layer with a set algorithm; Based on each of the probability values and the loss function generation formula, determine the current loss function value, and based on the current loss function value, adjust the learning parameters in the current text decoding model through backpropagation to obtain a text decoding model for the next iteration; Take the next iteration as the new current iteration and continue training the learning parameters until the iteration end condition is met to obtain a trained text decoding model.

[0243] According to one or more embodiments of the present disclosure, [Example XII] provides a text generation method. The loss function generation formula is expressed as: Take the logarithm of the sum of each of the probability values and take the negative of the logarithmic operation result.

[0244] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with (but not limited to) technical features with similar functions disclosed in the present disclosure.

[0245] In addition, although the operations are depicted in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0246] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. On the contrary, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A text generation method, characterized in that, comprising: inputting the obtained original text into a trained text encoding model to obtain text feature information; generating a target text corresponding to the original text based on the text feature information in combination with a trained text decoding model; wherein, the text decoding model includes a text prediction layer, the node information of a set number of nodes included in the text prediction layer is determined by the text feature information, and the target words included in the target text and the combination order of each target word are determined by the node information of each node and the topological structure between nodes; the text decoding model further includes: a position information input layer; the position information input layer includes a set number of node position parameters, and the set number is used to determine the number of nodes included in the text prediction layer; the text decoding model further includes: a basic decoding sub-model; the node information of a set number of nodes included in the text prediction layer is determined by each of the node position parameters and the text feature information in combination with the basic decoding sub-model, wherein each of the node position parameters and the text feature information serve as input data of the basic decoding sub-model, and the basic decoding sub-model outputs the node information of each node.

2. The method according to claim 1, characterized in that, the generating a target text corresponding to the original text based on the text feature information in combination with a trained text decoding model includes: inputting the text feature information and each of the node position parameters in the position information input layer into the basic decoding sub-model; obtaining the set number of initial text prediction vectors output by the basic decoding sub-model, and respectively using each of the initial text prediction vectors as the node information of each node in the text prediction layer; constructing a directed acyclic graph based on each of the nodes, determining the topological structure between nodes, and determining the target text of the original text in combination with the node information of each node.

3. The method according to claim 2, characterized in that, the constructing a directed acyclic graph based on each of the nodes, determining the topological structure between nodes, and determining the target text of the original text in combination with the node information of each node includes: constructing a directed acyclic graph according to the node labels of each node in the text prediction layer to obtain the topological structure between nodes; determining the node transition matrix corresponding to the text prediction layer according to the topological structure between nodes and the node information of each node; determining the target text of the original text according to the node information of each node and the node transition matrix.

4. The method according to claim 3, characterized in that, the determining the node transition matrix corresponding to the text prediction layer according to the topological structure between nodes and the node information of each node includes: for each node, determining the adjacent nodes to which the node is directed-connected from the topological structure between nodes; determining the transition probability from the node to each adjacent node according to the node and the node information of each adjacent node; forming the node transition matrix corresponding to the text prediction layer based on each of the transition probabilities.

5. The method according to claim 3, It is characterized in that determining the target text of the original text according to the node information of each node and the node transition matrix includes: According to the node information of each node, through the fully connected layer in the text prediction layer, determining the matching probability of each node to each word in the preset vocabulary; According to the node transition matrix and the matching probability of each node to each word, determining the predicted node and the corresponding target word; Based on each target word, combining to form the target text of the original text.

6. The method according to claim 5, It is characterized in that determining the predicted node and the corresponding target word according to the node transition matrix and the matching probability of each node to each word includes: Determining at least one predicted node according to the maximum transition probability corresponding to each node in the node transition matrix; For each predicted node, determining the maximum matching probability from the matching probabilities of the predicted node to each word, and determining the word corresponding to the maximum matching probability as the target word.

7. The method according to claim 5, It is characterized in that determining the predicted node and the corresponding target word according to the node transition matrix and the matching probability of each node to each word includes: Taking the node corresponding to the starting node label as the current node; Obtaining the current transition probability from the current node to each adjacent node from the node transition matrix; Determining the product value of each current transition probability and the matching probability corresponding to each word of the current node respectively; Selecting the maximum product value from each product value, and taking the adjacent node and word associated with the maximum product value as the predicted node and the target word respectively, and adding the predicted node and the target word to the cache table; Taking the predicted node as the new current node, and re-executing the selection operation of the current adjacent node corresponding to the current node until the loop end condition is reached.

8. The method according to claim 5, It is characterized in that determining the predicted node and the corresponding target word according to the node transition matrix and the matching probability of each node to each word includes: Based on the matching probability of each node to each word, determining the corresponding maximum matching probability, and determining the word corresponding to each maximum matching probability as the predicted word of the corresponding node; According to the pre-set path search algorithm, combining the node transition matrix and the predicted words of each node, determining the prediction path with the highest weight; Determining the predicted words corresponding to each predicted node in the prediction path as the corresponding target words.

9. The method according to any one of claims 1-8, It is characterized in that further comprising: Based on the set loss function generation strategy, training the learning parameters of the constructed text decoding model to obtain the trained text decoding model; Wherein, the learning parameters include: the node position parameters involved in the position information input layer included in the text decoding model, the basic model parameters involved in the basic decoding sub-model included, and the node-related parameters involved in each node included in the text prediction layer.

10. The method according to claim 9, It is characterized in that Based on the set loss function generation strategy, training the learning parameters of the constructed text decoding model to obtain the trained text decoding model, including: Obtaining at least one set of sample data, where one set of sample data includes an original sample text and a corresponding single target sample text; In the current iteration, after encoding the original sample text in a set of sample data using the text encoding model, inputting it into the current text decoding model; Based on the current text decoding model, determining the probability values corresponding to generating the target sample text from the original sample text through each text prediction path, where each text prediction path is formed by combining nodes in the text prediction layer with a set algorithm; Based on each of the probability values and the loss function generation formula, determining the current loss function value, and adjusting the learning parameters in the current text decoding model through backpropagation based on the current loss function value to obtain a text decoding model for the next iteration; Taking the next iteration as the new current iteration, continuing to train the learning parameters until the iteration end condition is met, and obtaining the trained text decoding model.

11. According to the method described in claim 10, the loss function generation formula is expressed as: Taking the logarithm of the sum of each of the probability values and taking the negative of the logarithmic operation result.

12. A text generation device Characterized in that It includes: An encoding execution module, configured to input the obtained original text into the trained text encoding model to obtain text feature information; A decoding execution module, configured to generate a target text corresponding to the original text based on the text feature information and in combination with the trained text decoding model; Wherein, the text decoding model includes a text prediction layer, the node information of the set number of nodes included in the text prediction layer is determined by the text feature information, and the target words included in the target text and the combination order of each target word are determined by the node information of each node and the topological structure between nodes; The text decoding model further includes: a position information input layer; The position information input layer includes a set number of node position parameters, and the set number is used to determine the number of nodes included in the text prediction layer; The text decoding model further includes: a basic decoding sub-model; The node information of the set number of nodes included in the text prediction layer is determined by each of the node position parameters and the text feature information in combination with the basic decoding sub-model, where each of the node position parameters and the text feature information serves as input data of the basic decoding sub-model, and the basic decoding sub-model outputs the node information of each node.

13. An electronic device Characterized in that The electronic device includes: One or more processors; A storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the text generation method described in any one of claims 1-11.

14. A computer-readable storage medium, on which a computer program is stored, Characterized in that When the program is executed by a processor, it implements the text generation method described in any one of claims 1-11.

Images