Text generation method, system and related device

Abstract

The invention discloses a text generation method and system and a related device, and the method comprises the steps: inputting a generated target text into a language model, and obtaining at least one first candidate character corresponding to an output layer of the current round of language model; wherein the first candidate character is obtained by using a reference network in the language model, and the reference network comprises a first decoding sub-network and a supplementary decoding layer in the language model; based on the first candidate character, obtaining a second candidate character corresponding to the first candidate character in the output layer of the current round; wherein the second candidate character uses a second decoding sub-network in the language model, the second decoding sub-network and the first decoding sub-network are stacked, and the second decoding sub-network and the supplementary decoding layer are arranged in parallel; and determining a target text corresponding to the current round based on the first candidate character and the second candidate character. Through the mode, the text generation efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text generation method, system and related devices. Background Art

[0002] With the continuous development of artificial intelligence, large-scale deep learning models have been widely used in different fields. However, as the model size increases, the computing requirements for model reasoning also grow exponentially, resulting in computing resource limitations and reasoning delays when the model generates matching text based on the user's query information or intent information in practical applications.

[0003] In view of this, how to improve the efficiency of text generation has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a text generation method, system and related devices, which can improve the efficiency of text generation.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a text generation method, comprising: inputting the generated target text into a language model, obtaining at least one first candidate character corresponding to the output layer of the language model of the current round; wherein the first candidate character is obtained by using a reference network in the language model, and the reference network includes a first decoding subnetwork and a supplementary decoding layer in the language model; based on the first candidate character, obtaining a second candidate character corresponding to the first candidate character in the output layer of the current round; wherein the second candidate character is obtained by using a second decoding subnetwork in the language model, and the second decoding subnetwork is stacked with the first decoding subnetwork, and the second decoding subnetwork is parallel to the supplementary decoding layer; based on the first candidate character and the second candidate character, determining the target text corresponding to the current round.

[0006] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide a text generation system, including: a first acquisition module, used to input the generated target text into a language model, and obtain at least one first candidate character corresponding to the output layer of the language model of the current round; wherein the first candidate character is obtained by using the reference network in the language model, and the reference network includes the first decoding subnetwork and the supplementary decoding layer in the language model; a second acquisition module, used to obtain the second candidate character corresponding to the first candidate character in the output layer of the current round based on the first candidate character; wherein the second candidate character is obtained by using the second decoding subnetwork in the language model, and the second decoding subnetwork is stacked with the first decoding subnetwork, and the second decoding subnetwork is parallel to the supplementary decoding layer; a processing module, used to determine the target text corresponding to the current round based on the first candidate character and the second candidate character.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising: a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the text generation method mentioned in the above technical solution.

[0008] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the text generation method mentioned in the above technical solution is implemented.

[0009] The beneficial effect of the present application is as follows: Different from the prior art, in the text generation method proposed in the present application, the generated target text is input into the reference network in the language model, and the first decoding subnetwork and the supplementary decoding layer in the reference network are used to perform at least one iterative process to obtain at least one first candidate character in the current round. Since the structure of the supplementary decoding layer in the reference network is simpler than that of the second decoding subnetwork in the language model, the computing resources consumed to obtain the first candidate character are less. In addition, the first candidate character is matched with the features obtained in the generation process, and the second decoding subnetwork and the output layer in the language model are used according to the features matched by all the first candidate characters in the current round, and the second candidate characters matching each first candidate character are obtained at the same time, and the first candidate characters are compared with the second candidate characters to obtain the target text with higher accuracy in the current round. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0011] Figure 1 It is a flowchart of an implementation method of the text generation method of the present application;

[0012] Figure 2 is a flowchart corresponding to an implementation method before step S101;

[0013] Figure 3 is a schematic diagram of the structure of the language model of the present application including the reference network;

[0014] Figure 4 is a flowchart of another implementation method corresponding to step S101;

[0015] Figure 5 is a flowchart of step S102 corresponding to another implementation manner;

[0016] Figure 6 is a flowchart of step S103 corresponding to another implementation manner;

[0017] Figure 7 is a flowchart of an implementation method corresponding to a training process of a reference network;

[0018] Figure 8 It is a structural schematic diagram of an implementation method of the application text generation system;

[0019] Fig. 9 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0020] Fig.10 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] See also Figure 1 , Figure 1It is a flow chart of an implementation method of the text generation method of the present application, the method comprising:

[0023] S101: Input the generated target text into the language model to obtain at least one first candidate character corresponding to the output layer of the current round language model; wherein the first candidate character is obtained by using a reference network in the language model, and the reference network includes a first decoding subnetwork and a supplementary decoding layer in the language model.

[0024] In one embodiment, in response to the trained language model having the ability to output matching text according to the target task, the target text generated according to the target task is input into the language model, and at least one first candidate character corresponding to the current round is obtained using the reference network in the language model and the output layer in the language model.

[0025] Specifically, the reference network includes a first decoding subnetwork and a supplementary decoding layer in a language model. The generated target text is input into the reference network, and the first decoding subnetwork, the supplementary decoding layer and the output layer in the language model in the reference network are iterated at least once to obtain at least one first candidate character corresponding to the current round.

[0026] In one implementation scenario, the target task includes but is not limited to intention information or inquiry information input by the user.

[0027] S102: Based on the first candidate character, obtain a second candidate character corresponding to the first candidate character in the output layer of the current round; wherein the second candidate character uses the second decoding subnetwork in the language model, and the second decoding subnetwork is stacked with the first decoding subnetwork, and the second decoding subnetwork is parallel to the supplementary decoding layer.

[0028] In one embodiment, in response to each first candidate character being generated, the first decoding subnetwork outputs a corresponding feature, and the feature is sequentially input into the second decoding subnetwork in the language model to simultaneously obtain the second candidate character corresponding to each first candidate character.

[0029] In another embodiment, in response to each first candidate character being generated, the first decoding subnetwork outputs a corresponding feature, the feature is concatenated, and the second candidate character corresponding to each first candidate character is simultaneously obtained based on the concatenated feature.

[0030] S103: Determine the target text corresponding to the current round based on the first candidate character and the second candidate character.

[0031] In one implementation, the first candidate characters are sequentially compared with the corresponding second candidate characters, and the target text corresponding to the current round is determined based on the comparison results.

[0032] Further, the target text corresponding to the current round is taken as the generated target text, and the process returns to step S101 and performs the subsequent steps in sequence until all the target texts corresponding to the target task are obtained. All the target texts are arranged in the order of generation as matching texts, and the matching texts are fed back to the user.

[0033] In one implementation scenario, in response to the target text obtained in the current round including a termination mark, it is considered that all target texts corresponding to the target task have been obtained.

[0034] In another embodiment, after all target texts corresponding to the current round are obtained, in response to all target texts having reached a maximum text length, it is considered that all target texts corresponding to the target task have been obtained, and after being arranged, they are fed back to the user as matching texts.

[0035] In the text generation method proposed in the present application, the generated target text is input into the reference network in the language model, and the first decoding subnetwork and the supplementary decoding layer in the reference network are used to perform at least one iterative process to obtain at least one first candidate character in the current round. Since the structure of the supplementary decoding layer in the reference network is simpler than that of the second decoding subnetwork in the language model, the computing resources consumed to obtain the first candidate character are less. In addition, the first candidate character matches the features obtained in the generation process, and the second decoding subnetwork and the output layer in the language model are used to match the features of all the first candidate characters in the current round, and the second candidate characters matching each first candidate character are obtained at the same time, and the first candidate characters are compared with the second candidate characters to obtain the target text with higher accuracy in the current round.

[0036] See also Figure 2 and Figure 3 , Figure 2 is a flowchart corresponding to an implementation method before step S101, Figure 3 1 is a schematic diagram of the structure of the language model of the reference network in the present application. The flow chart before step S101 corresponds to an implementation method. Specifically, the process before step S101 includes:

[0037] S201: Acquire a trained language model, wherein the language model includes a stacked first decoding subnetwork and a second decoding subnetwork.

[0038] In one embodiment, a trained language model is obtained, wherein the language model includes a plurality of stacked initial decoding layers, all of which are divided into a first decoding subnetwork and a second decoding subnetwork, and the first decoding subnetwork and the second decoding subnetwork respectively include a plurality of initial decoding layers adjacent to each other.

[0039] Specifically, in response to the language model including multiple initial decoding layers stacked in sequence, a target decoding layer is selected from all the initial decoding layers, and the target decoding layer and the initial decoding layer before the target decoding layer are constructed as a first decoding sub-network; and the initial decoding layer after the target decoding layer is constructed as a second decoding sub-network. Each initial decoding layer in the language model is used to extract features of different depths. The detailed structure of the initial decoding layer can refer to the decoder structure in the Transformer model, which will not be elaborated in detail here.

[0040] In addition, the above target decoding layer can be selected according to actual needs. However, it should be noted that the more the number of initial decoding layers in the first decoding sub-network is, the stronger its ability to understand the text is, but the more computing resources are needed.

[0041] For example, the initial decoding layers stacked in the language model include T 1 、T 2 、T 3 ,……,T i 、T i+1 ,……,T n , where T 1 is the first initial decoding layer, T n is the last initial decoding layer, and T is selected between the first initial decoding layer and the last initial decoding layer. i As the target decoding layer, and T 1 、T 2 、T 3 ,……,T i As the first decoding sub-network, T i+1 ,……,T n As the second decoding sub-network.

[0042] In one implementation scenario, the above-mentioned language model is a large language model with better data analysis capabilities, which may include but is not limited to deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTM), and generative pre-trained Transformer models, etc., and no specific restrictions are made on the specific construction and specific deployment of the large language model. In addition, the structure of the output layer in the language model includes a linear layer and a softmax layer.

[0043] In another embodiment, the target decoding layer is determined according to the application scenario of the language model. Specifically, a plurality of reference samples are obtained according to the application scenario of the language model, each reference sample includes a reference task and its corresponding matching text, and the target decoding layer is determined according to the average text length of the matching text corresponding to all reference samples. Among them, the longer the above average text length is, the more the number of initial decoding layers in the first decoding subnetwork is; that is, the longer the average text length is, the stronger the text comprehension ability required by the first decoding subnetwork is, and the initial decoding layer farther away from the first initial decoding layer is selected as the target decoding layer.

[0044] S202: Construct a reference network based on the first decoding subnetwork and the supplementary decoding layer in the language model.

[0045] In one embodiment, in order to improve the efficiency of text generation and save computing resources, a supplementary decoding layer is set in parallel with the second decoding subnetwork in the language model, and the first decoding subnetwork and the supplementary decoding layer are used as reference networks.

[0046] In one implementation scenario, the structure of the supplementary decoding layer is consistent with the structure of any initial decoding layer, and a reference network is constructed so that in actual application, the reference network processes the output of the first decoding subnetwork, and then the output layer of the language model outputs the corresponding result. Compared with directly using the second decoding subnetwork including multiple initial decoding layers to process the output of the first decoding subnetwork, this method consumes less computing resources and can output results faster.

[0047] In another embodiment, the supplementary decoding layer may also be a multi-layer structure, such as two or three layers, and each layer structure is consistent with any initial decoding layer structure. However, it should be noted that in order to improve the efficiency of text generation and save computing resources, the number of supplementary decoding layers should be less than the number of initial decoding layers in the second decoding subnetwork.

[0048] In addition, it should be noted that in order to improve the effect of the language model in practical applications, after setting the supplementary decoding layer, the reference network including the supplementary decoding layer needs to be trained. The specific training process will be described in the subsequent implementation method.

[0049] See also Figure 4 , Figure 4 is a flowchart corresponding to another implementation of step S101. Specifically, the implementation process of obtaining at least one first candidate character includes:

[0050] S301: Input the generated target text into the reference network, and use the first decoding subnetwork in the reference network to obtain the first candidate feature.

[0051] In one embodiment, in response to the first decoding subnetwork in the reference network including a word embedding layer and a plurality of stacked initial decoding layers, the generated target text is input into the word embedding layer to obtain a corresponding vector representation. Then, the initial decoding layer in the first decoding subnetwork sequentially understands and extracts the above vectors to obtain a first candidate feature for characterizing a deeper feature.

[0052] Among them, the process of understanding and extracting by the initial decoding layer in the first decoding subnetwork can refer to the working process of the decoder in the existing Transformer structure, and will not be elaborated here.

[0053] S302: Input the first candidate feature into a supplementary decoding layer in the reference network to obtain a second candidate feature.

[0054] In one embodiment, the first candidate feature obtained in step S301 is input into a supplementary decoding layer in the reference network, so as to use the supplementary decoding layer to perform deeper feature extraction on the first candidate feature to obtain a second candidate feature.

[0055] Among them, different from directly using multiple initial decoding layers in the second decoding sub-network to extract the first candidate features, the above method consumes less computational effort to obtain the second candidate features because the structure of the supplementary decoding layer is simpler.

[0056] S303: Based on the second candidate feature, obtain the first candidate character.

[0057] In one embodiment, after obtaining the second candidate feature, the second candidate feature is input into the output layer of the language model to obtain a vocabulary probability distribution output by the output layer. The vocabulary probability distribution includes multiple reference characters and a predicted probability matching each reference character.

[0058] Furthermore, the reference character corresponding to the maximum predicted probability in the vocabulary probability distribution is used as the first candidate character currently obtained.

[0059] S304: Use the first candidate characters to update the generated target text, return to the step of inputting the generated target text into the reference network, and use the first decoding subnetwork in the reference network to obtain the first candidate features, until all the first candidate characters corresponding to the current round are obtained.

[0060] In one embodiment, the generated target text is updated using the currently obtained first candidate character, and the process returns to step S301 to input the updated generated target text into the reference network, thereby obtaining the next first candidate character in the current round. In response to reaching the preset stop condition, all first candidate characters in the current round are obtained.

[0061] In one implementation scenario, the preset stop condition may be that the number of generated first candidate characters reaches a preset threshold, wherein the preset threshold may be flexibly set according to actual conditions.

[0062] In another embodiment, the above-mentioned preset stop condition can also be determined according to semantics. Specifically, in response to the first candidate character currently obtained being a semantic separation punctuation mark, such as a comma or a period, it is determined that the preset stop condition is satisfied.

[0063] See also Figure 5 , Figure 5 is a flowchart of another implementation method corresponding to step S102. Specifically, the implementation process of obtaining the second candidate character includes:

[0064] S401: Acquire a candidate feature sequence based on the first candidate features corresponding to all first candidate characters.

[0065] In one embodiment, in response to the first decoding subnetwork outputting the corresponding first candidate feature in the process of generating each first candidate character, based on the generation order of the first candidate characters in the current round, the first candidate features corresponding to all the first candidate characters are concatenated to obtain a candidate feature sequence.

[0066] S402: Input the candidate feature sequence into the second decoding sub-network to obtain the third candidate feature.

[0067] In one implementation, the obtained candidate feature sequence is input into a second decoding subnetwork in the language model to obtain a third candidate feature.

[0068] Specifically, after the candidate feature sequence is input into the second decoding subnetwork, each initial decoding layer in the second decoding subnetwork extracts the output result of the adjacent previous initial decoding layer in turn, and finally obtains the third candidate feature. The working process of the initial decoding layer in the second decoding subnetwork can refer to the working process of the decoder in the existing Transformer structure, and will not be elaborated here.

[0069] S403: Based on the third candidate feature, obtain the second candidate character corresponding to the output layer of the current round language model.

[0070] In one embodiment, the obtained third candidate features are input to an output layer in the language model, and the output layer performs classification according to the third candidate features and simultaneously outputs a second candidate character matching each first candidate character.

[0071] In the above steps, the candidate feature sequence is obtained by splicing the first candidate features matched by the first candidate characters in the generation process. The second decoding subnetwork is used to simultaneously output the second candidate characters corresponding to each first candidate character according to the candidate feature sequence, so that the first candidate characters are subsequently compared with the second candidate characters, which improves the accuracy of target text generation while consuming less computing resources.

[0072] See also Figure 6 , Figure 6 is a flowchart of another implementation method corresponding to step S103. Specifically, the implementation process of obtaining step S103 includes:

[0073] S501: Compare each first candidate character with the corresponding second candidate character in turn according to a target order to obtain a corresponding matching result, wherein the target order is consistent with the order in which the first candidate characters are obtained.

[0074] In one embodiment, current matching results between the first candidate character and the corresponding second candidate character are obtained in sequence according to the target order until the current matching result does not meet the preset matching condition.

[0075] Specifically, the generation order of each first candidate character in the current round is used as the target order, and the first candidate character is compared with the corresponding second candidate character in turn according to the target order to obtain the current matching result. In response to the current matching result corresponding to the current first candidate character meeting the preset matching condition, that is, the current first candidate character is the same as the corresponding second candidate character, the first candidate character or the second candidate character is considered to be an accurate output result, and the next first candidate character is judged. In response to the current matching result corresponding to the current first candidate character not meeting the preset matching condition, that is, the current first candidate character is not the same as the corresponding second candidate character, it is considered that there is a disagreement in the result output by the language model. In order to improve the accuracy, the next first candidate character is stopped from being compared with the corresponding second candidate character, that is, the current matching result between the subsequent first candidate character and the corresponding second candidate character is no longer obtained.

[0076] In another embodiment, each first candidate character may be compared with the corresponding second candidate character in turn according to the target order to obtain a matching result corresponding to each first candidate character.

[0077] S502: Based on the matching results, determine the target text corresponding to the current round.

[0078] In one embodiment, the current matching results between the first candidate characters and the corresponding second candidate characters are sequentially obtained according to the target order until the current matching results no longer meet the preset matching conditions. Then, based on the first candidate characters that meet the preset matching conditions, the target text corresponding to the current round is obtained, that is, the first candidate characters that meet the preset matching conditions are sorted according to the generated order to obtain the target text corresponding to the current round.

[0079] For example, the text obtained by arranging the multiple first candidate characters in the current round according to the generated order is: "Preventive measures are the focus of work", and the text composed of the corresponding second candidate characters is: "Preventive measures are the end point of work". After comparing them in sequence according to the target order, the first candidate characters "事", "前", "防", "范", "是", "工", "作" all meet the preset matching conditions, while the first candidate character "重" is inconsistent with the second candidate character "终", so the comparison of subsequent characters is no longer performed. Finally, the target text corresponding to the current round is "Preventive measures are work".

[0080] In another embodiment, after obtaining the matching results corresponding to each first candidate character according to the target order, all the first candidate characters before the first candidate character that does not meet the preset matching conditions are sorted according to the generated order to obtain the target text corresponding to the current round.

[0081] Please refer to Figure 7 , Figure 7 which is a schematic flowchart of the training process of the reference network corresponding to one embodiment. Specifically, the training process of the reference network includes:

[0082] S601: Obtain a plurality of training samples.

[0083] In one embodiment, a plurality of training samples for training the reference network are obtained.

[0084] Specifically, the training samples include training tasks and corresponding training matching texts.

[0085] In a specific application scenario, in response to the language model being a question-and-answer model for the medical field that can provide intelligent responses based on the questions input by users, the training samples include medical-related questions, and the corresponding training matching texts are the response texts given according to the questions.

[0086] In another embodiment, in response to the language model being trained using multiple samples, the samples used by the language model during training are directly used as the training samples of the reference network.

[0087] S602: Based on the training sample, obtain a first decoding feature output by a supplementary decoding layer in a reference network at a target round and a second decoding feature output by a second decoding subnetwork.

[0088] In one embodiment, the training sample is input into the language model to obtain the first decoding feature output by the supplementary decoding layer at the target round and the second decoding feature output by the second decoding subnetwork. In this embodiment, the specific acquisition process of the first decoding feature and the second decoding feature can refer to the acquisition process of the second candidate feature and the third candidate feature in the above corresponding embodiment, which will not be elaborated in detail here.

[0089] S603: Based on the first decoding feature and the second decoding feature, obtain the training loss, and use the training loss to adjust the parameters of the supplementary decoding layer until a preset convergence condition is met, and stop the training to obtain a trained reference network.

[0090] In one embodiment, a corresponding training loss is calculated based on the first decoding feature and the second decoding feature at the target round. The training loss is used to adjust the parameters in the supplementary decoding layer. In response to the obtained training loss converging or the training round reaching a threshold number, the training is stopped to obtain a trained reference network.

[0091] In one implementation scenario, the calculation function of the above training loss may be a mean square error loss function.

[0092] In another embodiment, the implementation process of step S603 may further include: after obtaining the first decoding feature and the second decoding feature at the target round, the first decoding feature and the second decoding feature are respectively input into the output layer of the language model to obtain the logits representations corresponding to the first decoding feature and the second decoding feature. The training loss is calculated using the mean square error loss function according to the logits representations corresponding to the first decoding feature and the second decoding feature, and the training loss is used to adjust the parameters in the supplementary decoding layer until the preset convergence condition is met to obtain a trained reference network.

[0093] In another embodiment, the implementation process of step S603 may further include: after obtaining the first decoding feature and the second decoding feature under the target round, the first decoding feature and the second decoding feature are respectively input into the output layer of the language model to obtain the vocabulary probability distribution corresponding to the first decoding feature and the second decoding feature respectively. The training loss is calculated using the KL divergence loss (Kullback-Leibler Divergence), and the training loss is used to adjust the 1 parameter in the supplementary decoding layer.

[0094] Through the above training process, the results obtained after the first decoding subnetwork and the supplementary decoding layer in the trained reference network are close to the results obtained after the first decoding subnetwork and the second decoding subnetwork in the language model. Therefore, the accuracy of the first candidate character obtained by the reference network in the above corresponding implementation method is higher.

[0095] See also Figure 8 , Figure 8 1 is a schematic diagram of the structure of an embodiment of the text generation system of the present application. Specifically, the text generation system comprises a first acquisition module 10, a second acquisition module 20 and a processing module 30 coupled to each other.

[0096] Specifically, the first acquisition module 10 is used to input the generated target text into the language model to obtain at least one first candidate character corresponding to the output layer of the current round language model; wherein the first candidate character is obtained using a reference network in the language model, and the reference network includes a first decoding subnetwork and a supplementary decoding layer in the language model.

[0097] The second acquisition module 20 is used to obtain the second candidate character corresponding to the first candidate character in the output layer of the current round based on the first candidate character; wherein the second candidate character is obtained using the second decoding subnetwork in the language model, and the second decoding subnetwork is stacked with the first decoding subnetwork, and the second decoding subnetwork is parallel to the supplementary decoding layer.

[0098] The processing module 30 is used to determine the target text corresponding to the current round based on the first candidate character and the second candidate character.

[0099] In one embodiment, the first acquisition module 10 inputs the generated target text into the language model to obtain at least one first candidate character corresponding to the output layer of the language model of the current round, including: inputting the generated target text into the reference network, and using the first decoding subnetwork in the reference network to obtain the first candidate feature; inputting the first candidate feature into the supplementary decoding layer in the reference network to obtain the second candidate feature; obtaining the first candidate character based on the second candidate feature; updating the generated target text with the first candidate character, returning to the step of inputting the generated target text into the reference network, and using the first decoding subnetwork in the reference network to obtain the first candidate feature, until all the first candidate characters corresponding to the current round are obtained.

[0100] In one embodiment, the second acquisition module 20 acquires the second candidate character corresponding to the first candidate character in the output layer of the current round based on the first candidate character, including: acquiring a candidate feature sequence based on the first candidate features corresponding to all the first candidate characters; inputting the candidate feature sequence into the second decoding subnetwork to acquire a third candidate feature; and acquiring the second candidate character corresponding to the output layer of the current round language model based on the third candidate feature.

[0101] In one embodiment, the processing module 30 determines the target text corresponding to the current round based on the first candidate character and the second candidate character, including: comparing each first candidate character with the corresponding second candidate character in turn according to the target order to obtain the corresponding matching result; wherein the target order is consistent with the order in which the first candidate characters are obtained; and determining the target text corresponding to the current round based on the matching result.

[0102] In one embodiment, the processing module 30 compares each first candidate character with the corresponding second candidate character in turn according to the target order to obtain a corresponding matching result, including: in response to the matching result corresponding to the current first candidate character not satisfying a preset matching condition, stopping comparing the next first candidate character with the corresponding second candidate character; the processing module 30 determines the target text corresponding to the current round based on the matching result, including: obtaining the target text corresponding to the current round based on the first candidate character that satisfies the preset matching condition.

[0103] In one embodiment, the language model includes a plurality of stacked initial decoding layers, all of which are divided into a first decoding subnetwork and a second decoding subnetwork, and the first decoding subnetwork and the second decoding subnetwork respectively include a plurality of initial decoding layers adjacent to each other; wherein the structure of the supplementary decoding layer is consistent with the structure of any initial decoding layer.

[0104] In one embodiment, please continue to refer to Figure 8 The text generation system proposed in the present application also includes a training module 40 coupled to the first acquisition module 10. The process of training the reference network by the training module 40 includes: obtaining multiple training samples; based on the training samples, obtaining the first decoding feature output by the supplementary decoding layer in the reference network under the target round, and the second decoding feature output by the second decoding subnetwork; based on the first decoding feature and the second decoding feature, obtaining the training loss, and using the training loss to adjust the parameters of the supplementary decoding layer until the preset convergence condition is met, stopping the training, and obtaining the trained reference network.

[0105] See also Fig. 9 , Fig. 9: It is a structural diagram of an embodiment of an electronic device of the present application. The electronic device includes: a memory 50 and a processor 60 coupled to each other. The memory 50 stores program instructions, and the processor 60 is used to execute the program instructions to implement the method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 60 can also be called a CPU (Center Processing Unit). The processor 60 may be an integrated circuit chip with signal processing capabilities. The processor 60 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 60 can be implemented by an integrated circuit chip.

[0106] See also Fig.10 , Fig.10 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 70 stores program instructions 80 that can be executed by a processor. When the program instructions 80 are executed by the processor, the method mentioned in any of the above embodiments is implemented.

[0107] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0111] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A text generation method, characterized in that: include: Input the generated target text into the language model to obtain at least one first candidate character corresponding to the output layer of the current round language model; wherein the first candidate character is obtained by using a reference network in the language model, and the reference network includes a first decoding subnetwork and a supplementary decoding layer in the language model; Based on the first candidate character, obtaining a second candidate character corresponding to the first candidate character in the output layer of the current round; wherein the second candidate character is obtained by using a second decoding subnetwork in the language model, and the second decoding subnetwork is stacked with the first decoding subnetwork, and the second decoding subnetwork is parallel to the supplementary decoding layer; Based on the first candidate character and the second candidate character, a target text corresponding to the current round is determined.

2. The method according to claim 1, characterized in that: The step of inputting the generated target text into the language model to obtain at least one first candidate character corresponding to the output layer of the current round language model includes: Inputting the generated target text into the reference network, and using the first decoding subnetwork in the reference network to obtain a first candidate feature; Inputting the first candidate feature into the supplementary decoding layer in the reference network to obtain a second candidate feature; Based on the second candidate feature, obtaining the first candidate character; The generated target text is updated using the first candidate characters, and the process returns to the step of inputting the generated target text into the reference network, and obtaining the first candidate features using the first decoding subnetwork in the reference network, until all the first candidate characters corresponding to the current round are obtained.

3. The method according to claim 2, characterized in that The acquiring, based on the first candidate character, a second candidate character corresponding to the first candidate character in the output layer of the current round includes: Based on the first candidate features corresponding to all the first candidate characters, obtaining a candidate feature sequence; Inputting the candidate feature sequence into the second decoding subnetwork to obtain a third candidate feature; Based on the third candidate feature, a second candidate character corresponding to the output layer of the language model in the current round is obtained.

4. The method according to claim 1, characterized in that: The determining the target text corresponding to the current round based on the first candidate character and the second candidate character includes: Comparing each of the first candidate characters with the corresponding second candidate characters in sequence according to a target order to obtain a corresponding matching result; wherein the target order is consistent with the order in which the first candidate characters are obtained; Based on the matching result, the target text corresponding to the current round is determined.

5. The method according to claim 4, characterized in that The step of sequentially comparing each of the first candidate characters with the corresponding second candidate characters according to the target order to obtain a corresponding matching result includes: In response to the matching result corresponding to the current first candidate character not satisfying the preset matching condition, stopping comparing the next first candidate character with the corresponding second candidate character; The determining the target text corresponding to the current round based on the matching result includes: Based on the first candidate character that meets the preset matching condition, the target text corresponding to the current round is obtained.

6. The method according to claim 1, characterized in that The language model includes a plurality of stacked initial decoding layers, all of the initial decoding layers are divided into the first decoding sub-network and the second decoding sub-network, and the first decoding sub-network and the second decoding sub-network respectively include a plurality of the initial decoding layers adjacent to each other; The structure of the supplementary decoding layer is consistent with any of the initial decoding layer structures.

7. The method according to claim 1, characterized in that The training method of the reference network comprises: Obtain multiple training samples; Based on the training sample, obtaining a first decoding feature output by a supplementary decoding layer in the reference network at a target round and a second decoding feature output by the second decoding subnetwork; Based on the first decoding feature and the second decoding feature, a training loss is obtained, and the parameters of the supplementary decoding layer are adjusted using the training loss until a preset convergence condition is met, and the training is stopped to obtain the trained reference network.

8. A text generation system, characterized in that: include: A first acquisition module is configured to input the generated target text into a language model to acquire at least one first candidate character corresponding to an output layer of the current round of the language model; wherein the first candidate character is obtained by using a reference network in the language model, and the reference network includes a first decoding subnetwork and a supplementary decoding layer in the language model; A second acquisition module is used to acquire, based on the first candidate character, a second candidate character corresponding to the first candidate character in the output layer of the current round; wherein the second candidate character is obtained by using a second decoding subnetwork in the language model, and the second decoding subnetwork is stacked with the first decoding subnetwork, and the second decoding subnetwork is parallel to the supplementary decoding layer; A processing module is used to determine a target text corresponding to a current round based on the first candidate character and the second candidate character.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the text generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the text generation method according to any one of claims 1 to 7 is implemented.

Images