A method and apparatus for training a text generation model

Abstract

Embodiments of the present application provide a text generation model training method and device. The method provided by the present application comprises: obtaining a first neural network model; adjusting the input relationship of an encoder in the first neural network model to obtain a second neural network model, wherein the Lth encoder in the second neural network model is used to receive the feature vectors output by the L-Mth encoder; training the second neural network model using training data to obtain a third neural network model; and sending the third neural network model to a target device. The method provided by the present application can improve the speed of generating text.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a training method and apparatus for a text generation model. Background Technology

[0002] Text generation is a crucial research task in the field of natural language processing, aiming to automatically generate relevant descriptive text based on input data. For example, the input data can be any of the following: tabular data, Structured Query Language (SQL) data, and logical data; or it can be a combination of tabular data and SQL data.

[0003] Currently, text generation is typically based on end-to-end generative models. An end-to-end model can be considered a black box model; given graph-structured data (input data), this black box model can generate text information corresponding to that input data. Generally, when generating text using end-to-end generative models, the designed end-to-end generative model needs to be trained using training data, and then the trained model is used to generate text.

[0004] However, text generation based on currently trained models is slow. Therefore, improving the speed of text generation has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a training method and apparatus for a text generation model. The end-to-end generation model trained by this method can improve the speed of text generation during the inference stage.

[0006] In a first aspect, this application provides a method for training a text generation model, comprising: acquiring a first neural network model, the first neural network model including N1 encoders and a target decoder, wherein the N1 encoders have identical structures, the first encoder among the N1 encoders is used to receive graph structure data, perform feature extraction on the graph structure data, and output the extracted feature vector, the i-th encoder among the N1 encoders is used to receive the feature vector output by the (i-1)-th encoder, perform feature extraction on the received feature vector, and output the extracted feature vector, and the target decoder is used to receive the feature vector output by the N1-th encoder among the N1 encoders, generate text based on the received feature vector, and output the generated text, where N1 is an integer greater than 1, and i is an integer ranging from 2 to N1; adjusting the input relationship of the encoders in the first neural network model to obtain a second neural network model, wherein the L-th encoder in the second neural network model is used to receive the feature vector output by the LM-th encoder; training the second neural network model using training data to obtain a third neural network model; and sending the third neural network model to a target device.

[0007] It is understood that in this embodiment, when training the first neural network, the input relationship of the first neural network model is first adjusted to obtain the second neural network model, and then the second neural network model is trained.

[0008] It should be understood that since the Lth encoder in the second neural network model receives the feature vector output by the LMth encoder, the M encoders between the LMth and Lth encoders are not involved in the training when the second neural network model is trained using training data. That is, the trained second neural network model can also be understood as a sub-model of the first neural network model.

[0009] It should be noted that the above only describes a training method for training a text generation model once. Typically, in practice, after obtaining a first neural network model including multiple encoders, this model can be trained K times. During each training iteration, the input relationships of the encoders are adjusted; that is, not all encoders in the first neural network model participate in training each time. Understandably, the final parameters of the first neural network model trained in this way ensure that removing any one encoder will not affect the final generation result. In this case, when using the neural network model obtained by the training method of this application for inference, it is possible to generate text using a model that has removed several encoders from the trained neural network model, thereby reducing the complexity of the neural network model and improving the speed of text generation.

[0010] In conjunction with the first aspect, in one possible implementation, the target device is a second server.

[0011] In conjunction with the first aspect, in one possible implementation, the target device is a terminal device.

[0012] In conjunction with the first aspect, in one possible implementation, the M encoders between the Lth encoder and the LMth encoder are randomly selected encoders.

[0013] In conjunction with the first aspect, in one possible implementation, the method further includes: obtaining the weight of each encoder among the N1 encoders; and determining the encoders corresponding to the first M weights after arranging the N1 weights of the N1 encoders in ascending order as the M encoders between the Lth encoder and the LMth encoder.

[0014] In conjunction with the first aspect, in one possible implementation, the method further includes: acquiring first structured data; processing the first structured data using a preset algorithm to obtain first graph structured data, wherein the preset algorithm is used to convert the structured data into graph structured data; and determining the first graph structured data and the target text information corresponding to the first structured data as the training data.

[0015] Secondly, this application provides a text generation method, comprising: acquiring graph structure data to be processed; and using a target neural network model to obtain text information based on the graph structure data to be processed, wherein the target neural network model is a neural network model trained using the method of the first aspect or any one thereof.

[0016] In this embodiment, a target neural network model is used to generate text from the graph-structured data to be processed. The target neural network model is trained as follows: during each training phase, not all encoders are included in the training, ensuring that the final parameters of the trained neural network model are such that removing any encoder does not affect the final generation effect. Therefore, when using the target neural network model obtained through this training method to generate text from the graph-structured data, one or more encoders can be removed, reducing the computational load and thus improving the speed of text generation from the graph-structured data.

[0017] In conjunction with the second aspect, in one possible implementation, the method further includes: receiving the target neural network model sent by the first server.

[0018] Thirdly, this application provides a training apparatus for a text generation model, comprising: an acquisition module for acquiring a first neural network model, the first neural network model including N1 encoders and a target decoder, the N1 encoders having identical structures, the first encoder among the N1 encoders being used to receive graph structure data, perform feature extraction on the graph structure data, and output the extracted feature vector, the i-th encoder among the N1 encoders being used to receive the feature vector output by the (i-1)-th encoder, perform feature extraction on the received feature vector, and output the extracted feature vector, the target decoder being used to receive the feature vector output by the N1-th encoder among the N1 encoders, generate text based on the received feature vector, and output the generated text, where N1 is an integer greater than 1, and i is an integer ranging from 2 to N1; a processing module for adjusting the input relationship of the encoders in the first neural network model to obtain a second neural network model, the L-th encoder in the second neural network model being used to receive the feature vector output by the LM-th encoder; the processing module is further used to train the second neural network model using training data to obtain a third neural network model; the processing module is further used to send the third neural network model to a target device.

[0019] In conjunction with the third aspect, in one possible implementation, the target device is a second server.

[0020] In conjunction with the third aspect, in one possible implementation, the target device is a terminal device.

[0021] In conjunction with the third aspect, in one possible implementation, the M encoders between the Lth encoder and the LMth encoder are randomly selected encoders.

[0022] In conjunction with the third aspect, in one possible implementation, the acquisition module is further configured to: acquire the weight of each encoder among the N1 encoders; the processing module is further configured to: determine the encoders corresponding to the first M weights after arranging the N1 weights corresponding to the N1 encoders in ascending order as the M encoders between the Lth encoder and the LMth encoder.

[0023] Fourthly, this application provides a training apparatus for a text generation model, including a processor for calling a computer program from memory, wherein when the computer program is executed, the processor is used to perform the method described in the first aspect or any possible implementation thereof.

[0024] Fifthly, this application provides a text generation apparatus, including a processor configured to call a computer program from a memory, wherein when the computer program is executed, the processor is configured to perform the method described in the second aspect or any possible implementation thereof.

[0025] In a sixth aspect, this application provides a computer-readable storage medium for storing a computer program, the computer program including code for performing the methods described in the first aspect or any possible implementation thereof. Attached Figure Description

[0026] Figure 1 This is a structural schematic diagram of an application scenario provided in an embodiment of this application;

[0027] Figure 2 A flowchart illustrating a training method for a text generation model provided in an embodiment of this application;

[0028] Figure 3 A structural schematic diagram of the first neural network model provided in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram illustrating the conversion of first structured data into a graph structure according to one embodiment of this application;

[0030] Figure 5 A schematic diagram illustrating the conversion of first structured data into a graph structure, provided as another embodiment of this application;

[0031] Figure 6 This is a schematic diagram of the encoder structure provided in one embodiment of this application;

[0032] Figure 7 This is a schematic diagram of the structure of a training device for a text generation model provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of the structure of a training device for a text generation model provided in another embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] To facilitate understanding, several terms involved in the embodiments of this application will be introduced first.

[0036] 1. Text generation

[0037] Text generation is a crucial yet challenging task in the field of natural language processing, aiming to transform input data (e.g., sequences and keywords) into coherent and readable text in a natural language model. Its goal is to generate readable natural language text, with representative applications including dialogue systems, text summarization, and machine translation.

[0038] 2. Table-based question-and-answer technique

[0039] Table-based question answering is designed for natural language (NL) questions, using NL technology to provide answers based on the content of a table of data. In table-based question answering scenarios such as product information inquiries, the table data is typically in the hundreds of thousands or even millions, representing a very large data volume.

[0040] Specifically, table-based question answering is a question-answering technology based on relational data storage. This technology first converts the user's natural language into a structured query language (SQL) statement, then retrieves the answer information from the stored database table based on the SQL statement, and finally generates corresponding response information based on the SQL statement and the answer information to answer the user's question.

[0041] 3. Neural Networks

[0042] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0043] The key technology in artificial intelligence today is neural networks (NN). Neural networks connect a large number of simple processing units (called neurons) to form complex network systems by simulating the connections between nerve cells in the human brain.

[0044] A simple neural network consists of three layers: an input layer, an output layer, and hidden layers (also called intermediate layers). Each connection between layers corresponds to a weight (the value of which is called a weight or parameter). The reason why neural networks perform well in fields such as computer vision and natural language processing is that the weights are adjusted through training algorithms to optimize the prediction results of the neural network.

[0045] The training of a neural network generally involves two computational steps: forward computation and backward computation. Forward computation involves calculating the input values ​​and parameters, then passing them through a non-linear function to produce the output value. This output value either serves as the network's final output or is used as input for subsequent similar calculations. The deviation between the network's output value and the actual label value of the corresponding sample is measured by a loss function, f(x,w), which is a function of the input sample x and the network parameters w. To minimize the loss function, the network parameters w need to be continuously adjusted. Backward computation is used to obtain updated values ​​for the parameters w. In gradient descent-based algorithms, backward computation starts from the last layer of the neural network, calculating the partial derivatives of the loss function with respect to the parameters of each layer, ultimately obtaining the partial derivatives of all parameters, known as the gradient. In each iteration, the parameters w are updated in the opposite direction of the gradient with a step size η, resulting in new parameters w, thus completing one training step. This update process is represented by the following equation:

[0046]

[0047] Among them, w t w represents the parameter used in the t-th iteration. t+1 This represents the updated parameters, where η is called the learning rate, and B... t Let represent the set of samples input in the t-th iteration.

[0048] The process of training a neural network is the process of learning the weights corresponding to neurons. Its ultimate goal is to obtain the weights corresponding to each layer of neurons in the trained neural network.

[0049] In recent years, with the rapid development of artificial intelligence technology, text generation has become an important technique in natural language processing. Its goal is to automatically generate relevant descriptive text based on input data. For example, the input data can be any of the following: tabular data, Structured Query Language (SQL) data, and logical data, or a combination of both. It should be noted that the concepts of tabular data, SQL data, and logical data can be found in relevant technical documentation and will not be elaborated upon here.

[0050] For example, Figure 1 This is a structural diagram illustrating an application scenario provided in one embodiment of this application. For example... Figure 1As shown, in this application scenario, the training server 101 can use a classification algorithm to train the structured data in the structured database to obtain a trained model. Then, the training server 101 can send the trained model to the target device 102 so that the target device 102 can use the trained model to generate the text information corresponding to the new sample data when it receives new sample data.

[0051] It should be noted that the specific type of generation task in this application scenario is not limited in this embodiment. For example, the text generation system can perform a task based on table data to text, also known as Table-to-Text generation; or, it can perform a task based on structured query language (SQL) data to text, also known as SQL-to-Text generation; or, it can perform a task based on logic data to text, also known as Logic-to-Text generation; or, it can also perform a task based on response natural language generation (ResponseNLG) data to text. It should be understood that when the text generation system 100 performs the above tasks, it should first preprocess the corresponding data into graph structure data.

[0052] Usually, in Figure 1 In the application scenario shown, the preset model included in the training server 101 is an end-to-end generative model. This end-to-end model can be considered a black-box model; given data represented by a graph structure (input data), this black-box model can generate text information corresponding to that input data. Typically, when generating text based on an end-to-end generative model, it is necessary to train the designed end-to-end generative model using training data, and then use the trained model to generate text after training is complete.

[0053] However, text generation based on currently trained models is slow. Therefore, improving the speed of text generation has become an urgent technical problem to be solved.

[0054] In view of this, embodiments of this application provide a training method and apparatus for a text generation model. In the training method for the text generation model provided by this application, after obtaining a first neural network model including multiple encoders, during the training process using training data, a sub-model is determined from the first neural network model at each training iteration. That is, not all encoders participate in the training of the first neural network model at each iteration. It is understood that the final parameters of the first neural network model trained in this way can ensure that removing any encoder from the first neural network model will not affect the final generation effect. In this case, when using the neural network model obtained by the training method of this application for inference, several layers of the trained neural network model can be removed to generate text, thereby reducing the complexity of the model and further improving the speed of text generation.

[0055] Figure 2 This is a flowchart illustrating the training method of the text generation model according to an embodiment of this application. Figure 2 As shown, the method in this embodiment may include S201, S202, S203, and S204, and the method may be performed by... Figure 1 The training server 101 shown is executing.

[0056] It should be noted that in this embodiment, the training server 101 is also referred to as the first server.

[0057] S201, Obtain the first neural network model.

[0058] It should be understood that if you want to train a neural network model for a specific text generation task, the first step should be to obtain a pre-designed neural network model, which includes a large number of parameters that need to be trained; then, use training data related to the specific task to adjust the pre-designed neural network model to determine the appropriate parameters, thereby completing the training process.

[0059] In this embodiment, the first neural network model includes N1 encoders and a target decoder. The N1 encoders have identical structures.

[0060] Specifically, Figure 3 This is a structural schematic diagram of the first neural network model provided in the embodiments of this application, such as... Figure 3As shown, the first encoder among the N1 encoders receives graph structure data, performs feature extraction on the graph structure data, and outputs the extracted feature vector. Any encoder from the second encoder to the N1th encoder receives the feature vector output by the previous encoder, and continues to perform feature extraction on the received feature vector and outputs the extracted feature vector.

[0061] like Figure 3 As shown, for the target decoder, it receives the feature vector output from the N1th encoder, and generates and outputs the text based on the feature vector output from the N1th encoder.

[0062] It should be noted here that... Figure 3 This is merely an example of a target decoder and does not constitute a limitation of this application. For example, the number of target decoders can be N2. When the number of target decoders is N2, the first decoder among the N2 target decoders receives the information output by the N1st encoder, and either the second target decoder or the N2nd decoder receives the output information of the previous target decoder. The N2nd target decoder outputs text information based on the output information of the N2-1th target decoder.

[0063] S202, the input relationship of the encoder in the first neural network model is adjusted to obtain the second neural network model. The Lth encoder in the second neural network model is used to receive the feature vector output by the LMth encoder.

[0064] It should be understood that currently, when training the first neural network model, the input relationships of the first neural network model remain unchanged. That is, for any encoder, its input information is fixed. However, in this embodiment, when training the first neural network model, the input relationships of the encoders in the first neural network model are changed first to obtain the second neural network model.

[0065] Specifically, in this embodiment, after changing the input relationship, the Lth encoder originally received the feature vector output by the (L-1)th encoder, but now it receives the feature vector output by the LMth encoder.

[0066] For ease of understanding, Figure 3 The first neural network model shown is an example. Figure 3 As shown, in traditional methods, each time the first neural network model is trained, the trained neural network model must include all encoders. However, in this scheme, the input relationships are adjusted before each training iteration; for example, for... Figure 3 The first neural network model shown can directly input the feature vector output by encoder 1 into encoder 3, bypassing encoder 2.

[0067] S203, Use the training data to train the second neural network model to obtain the third neural network model.

[0068] As can be seen from S202, the Lth encoder in the second neural network model receives the feature vector output by the LMth encoder. Therefore, when training the second neural network model using training data, it is equivalent to the M encoders between the LMth and Lth encoders not participating in the training. That is, the trained second neural network model can also be understood as a sub-model of the first neural network model.

[0069] Understandably, after training the second neural network model using training data, the parameters in the second neural network model will be adjusted to obtain the third neural network model.

[0070] S204, send the third neural network model to the target device.

[0071] For example, the target device is a server that can deploy neural network models (also referred to as a second server in this embodiment), or a terminal device that can deploy neural network models.

[0072] It should be understood that when the target device is the second server, if a third neural network model is to be used to generate text information, the terminal device can first send the structured data for which text information needs to be generated (e.g., new sample data) to the second server, and then the second server generates the text information for that new sample data. That is, in this scenario, the terminal device can obtain the text information corresponding to the new sample data by accessing the second server.

[0073] It should be understood that when the target device is a terminal device, if a third neural network model is to be used to generate text information, the terminal device can directly input the structured data (e.g., new sample data) into the target neural network model to generate the text information, and then generate the text information of the new sample data.

[0074] It should be noted that the above only describes a training method for training a text generation model once. In practice, after obtaining a first neural network model with multiple encoders, this model can be trained multiple times. During each training iteration, the input relationships of the encoders are adjusted; that is, not all encoders in the first neural network model participate in the training each time. Understandably, the final parameters of the first neural network model trained in this way ensure that removing any one encoder will not affect the final generation result.

[0075] The above explains how this application trains a text generation model. It is understood that in a specific text generation task, a pre-designed neural network model can be trained based on training data specific to that task to obtain a target neural network model for that task. This target neural network model can then be used for prediction (also known as inference). Specifically, the prediction process includes: acquiring graph structure data to be processed; and using the target neural network model to obtain text information based on the graph structure data to be processed.

[0076] Understandably, since the target neural network model trained in this embodiment can guarantee that removing any encoder will not affect the final generation effect, when using the target neural network model to generate text from graph-structured data, one or more encoders in the target neural network model can be pruned to reduce the computational load when using the target neural network model to generate text from graph-structured data, thereby improving the speed of text generation from graph-structured data using the target neural network model.

[0077] It should be noted that, in this embodiment, pruning one or more encoders in the target neural network model refers to changing the input relationship of the encoders in the target neural network model. For example, in a target neural network with 10 encoders, the 5th encoder can be pruned, and pruning the 5th encoder means inputting the feature vector output by the 4th encoder into the 6th encoder.

[0078] One possible implementation is to pre-set the number of encoders to be pruned, and then perform random pruning. Alternatively, each encoder can be sorted by weight first, and then encoders with smaller weights can be pruned.

[0079] In this case, when the neural network model obtained by the training method of this application is used for inference, it is possible to generate text by removing several encoders from the trained neural network model, thereby reducing the complexity of the neural network model and improving the speed of text generation.

[0080] It should be noted that the embodiments of this application do not limit the specific implementation method of how to adjust the input relationship of the encoder in the first neural network model to obtain the second neural network model.

[0081] For example, the M encoders between the Lth encoder and the LMth encoder are randomly selected encoders.

[0082] For example, the M encoders between the Lth encoder and the LMth encoder are determined based on the weight of each encoder. In this case, the method in this embodiment also needs to include the following steps: 1) obtaining the weight of each encoder among the N encoders; 2) determining the encoders corresponding to the first M weights of the N1 encoders arranged in ascending order as the M encoders.

[0083] It should be understood that when the first neural network model used in this embodiment is a graph-to-text model, it is also called a graph-to-text model. Therefore, if the data used to train the first neural network model is not graph-structured data, but structured data (referred to as first structured data), then it is necessary to preprocess the first structured data to convert it into graph-structured data.

[0084] For example, when the first structured data is SQL data: Select Fixed Increase Year, Increase Purpose, Where Latest Price < '10', Figure 4 This is a schematic diagram illustrating the conversion of first structured data into a graph structure, as provided in one embodiment of this application. Figure 4 As shown, the first node in the graph structure is the root node. The root node is then connected to the Select node and the Where node. The Select node is connected to two "AGG:none" nodes. One "AGG:none" node is connected to the fixed increase year node, and the other "AGG:none" node is connected to the issuance destination node. The Where node is connected to the operation "Op:<" node. The "Op:<" node is connected to the latest price node and 10.

[0085] For example, when the first structured data is the Table data shown in Table 1:

[0086] Table 1 Data

[0087] Private Placement Year Purpose of additional issuance 2016 Financing other assets 2016 Project financing

[0088] Figure 5 This is a schematic diagram illustrating the conversion of first structured data into a graph structure, as provided in another embodiment of this application. Figure 5 As shown, the transformed graph structure includes 5 nodes. Among them, the node "Private Placement Year" is connected to node "2016", the node "Private Placement Purpose" is connected to node "Other Assets Financing" and node "Project Financing", node "2016" is connected to node "Other Assets Financing" and node "Project Financing", and node "Other Assets Financing" is connected to node "Project Financing".

[0089] It should be noted here that... Figure 4 and Figure 5The graph structure shown is merely an example and does not constitute a limitation of this application.

[0090] Based on the above embodiments, it should be noted that each encoder in the embodiments of this application may further include more structures. For example... Figure 6 As shown, the encoder consists of N1 encoder modules. Each encoder module includes a transfer coding block (transformer encoder block), followed by a regularization module, and then a graph attention network module. The concepts of the transformer encoder block, regularization module, and graph attention network module can be found in relevant technical descriptions and will not be elaborated upon here.

[0091] Figure 7 This is a schematic diagram of the structure of a training device 700 for a text generation model provided in one embodiment of this application. Figure 7 As shown, the device 700 includes an acquisition module 701 and a processing module 702.

[0092] The acquisition module 701 is used to acquire a first neural network model, which includes N1 encoders and a target decoder. The N1 encoders have the same structure. The first encoder among the N1 encoders is used to receive graph structure data, extract features from the graph structure data, and output the extracted feature vector. The i-th encoder among the N1 encoders is used to receive the feature vector output by the (i-1)-th encoder, extract features from the received feature vector, and output the extracted feature vector. The target decoder is used to receive the feature vector output by the N1-th encoder among the N1 encoders, generate text based on the received feature vector, and output the generated text. N1 is an integer greater than 1, and i is an integer ranging from 2 to N1. The processing module 702 is used to adjust the input relationship of the encoders in the first neural network model to obtain a second neural network model. The L-th encoder in the second neural network model is used to receive the feature vector output by the LM-th encoder. The processing module 702 is also used to train the second neural network model using training data to obtain a third neural network model. The processing module 702 is also used to train the second neural network model using training data to obtain the third neural network model.

[0093] In one possible implementation, the target device is a second server.

[0094] In one possible implementation, the target device is a terminal device.

[0095] In one possible implementation, the M encoders between the Lth encoder and the LMth encoder are randomly selected encoders.

[0096] In one possible implementation, the acquisition module 701 is further configured to: acquire the weight of each encoder among the N1 encoders; the processing module 702 is further configured to: determine the encoders corresponding to the first M weights after arranging the N1 weights corresponding to the N1 encoders in ascending order as the M encoders between the Lth encoder and the LMth encoder.

[0097] In one possible implementation, the acquisition module 701 is further configured to: acquire first structured data; the processing module 702 is further configured to: process the first structured data using a preset algorithm to obtain first graph structured data, wherein the preset algorithm is used to convert the structured data into graph structured data; the processing module 702 is further configured to: determine the first graph structured data and the target text information corresponding to the first structured data as the training data.

[0098] Figure 8 This is a schematic diagram of the structure of a training device 800 for a text generation model according to an embodiment of this application. The device 800 is used to perform the methods described above.

[0099] The device 800 includes a processor 810, which is configured to execute computer programs or instructions stored in the memory 820, or to read data stored in the memory 820, to perform the methods described in the above method embodiments. Optionally, there may be one or more processors 810.

[0100] Optionally, such as Figure 8 As shown, the device 800 also includes a memory 820 for storing computer programs or instructions and / or data. The memory 820 may be integrated with the processor 810 or may be disposed separately. Optionally, there may be one or more memories 820.

[0101] Optionally, such as Figure 8 As shown, the device 800 also includes a communication interface 830, which is used for receiving and / or transmitting signals. For example, the processor 810 is used to control the communication interface 830 to receive and / or transmit signals.

[0102] Optionally, the device 800 is used to implement the operations described in the various method embodiments above.

[0103] For example, processor 810 is used to execute computer programs or instructions stored in memory 820 to implement the relevant operations of the various method embodiments described above. For example, processor 810 can be used to: acquire a first neural network model, the first neural network model including N1 encoders and a target decoder, the N1 encoders having the same structure, the first encoder of the N1 encoders being used to receive graph structure data, perform feature extraction on the graph structure data and output the extracted feature vector, the i-th encoder of the N1 encoders being used to receive the feature vector output by the (i-1)-th encoder, perform feature extraction on the received feature vector and output the extracted feature vector, the target decoder being used to receive the feature vector output by the N1-th encoder of the N1 encoders, generate text based on the received feature vector and output the generated text, where N1 is an integer greater than 1, and i is an integer ranging from 2 to N1; adjust the input relationship of the encoders in the first neural network model to obtain a second neural network model, the L-th encoder of the second neural network model being used to receive the feature vector output by the LM-th encoder; train the second neural network model using training data to obtain a third neural network model; and send the third neural network model to a target device.

[0104] In one example, the target device is a second server.

[0105] In one example, the target device is a terminal device.

[0106] In one example, the M encoders between the Lth encoder and the LMth encoder are randomly selected encoders.

[0107] In some examples, the processor 810 is also configured to: obtain the weight of each encoder among the N1 encoders; and determine the encoders corresponding to the first M weights after arranging the N1 weights of the N1 encoders in ascending order as the M encoders between the Lth encoder and the LMth encoder.

[0108] In some examples, the processor 810 is further configured to: acquire first structured data; process the first structured data using a preset algorithm to obtain first graph structured data, wherein the preset algorithm is used to convert the structured data into graph structured data; and determine the first graph structured data and the target text information corresponding to the first structured data as the training data.

[0109] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU (which can be understood as a type of microprocessor), or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0110] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0111] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0112] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement... Figure 2 The steps in the method described.

[0113] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement... Figure 2 The steps in the method described.

[0114] Accordingly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, cause the processor to implement... Figure 2 The steps in the method described.

Claims

1. A training method for a text generation model, characterized in that, Applied to the first server, including: A first neural network model is obtained, comprising N1 encoders and a target decoder. The N1 encoders have the same structure. The first encoder among the N1 encoders is used to receive graph structure data, perform feature extraction on the graph structure data, and output the extracted feature vector. The i-th encoder among the N1 encoders is used to receive the feature vector output by the (i-1)-th encoder, perform feature extraction on the received feature vector, and output the extracted feature vector. The target decoder is used to receive the feature vector output by the N1-th encoder among the N1 encoders, generate text based on the received feature vector, and output the generated text. N1 is an integer greater than 1, and i is an integer ranging from 2 to N1. The input relationship of the encoder in the first neural network model is adjusted to obtain the second neural network model. The Lth encoder in the second neural network model is used to receive the feature vector output by the LMth encoder. The second neural network model is trained using the training data to obtain the third neural network model; The third neural network model is sent to the target device; Obtain the weight of each encoder among the N1 encoders; The encoders corresponding to the first M weights of the N1 encoders arranged in ascending order are determined as the M encoders between the Lth encoder and the LMth encoder.

2. The method according to claim 1, characterized in that, The target device is the second server.

3. The method according to claim 1, characterized in that, The target device is a terminal device.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain the first structured data; The first structured data is processed using a preset algorithm to obtain the first graph structure data. The preset algorithm is used to convert the structured data into graph structure data. The first graph structure data and the target text information corresponding to the first structure data are determined as the training data.

5. A text generation method, characterized in that, include: Obtain the graph structure data to be processed; Text information is obtained using a target neural network model based on the graph structure data to be processed, wherein the target neural network model is a neural network model trained using any one of claims 1 to 4.

6. The method according to claim 5, characterized in that, The method further includes: Receive the target neural network model sent by the first server.

7. A training device for a text generation model, characterized in that, include: An acquisition module is used to acquire a first neural network model, which includes N1 encoders and a target decoder. The N1 encoders have the same structure. The first encoder among the N1 encoders is used to receive graph structure data, perform feature extraction on the graph structure data, and output the extracted feature vector. The i-th encoder among the N1 encoders is used to receive the feature vector output by the (i-1)-th encoder, perform feature extraction on the received feature vector, and output the extracted feature vector. The target decoder is used to receive the feature vector output by the N1-th encoder among the N1 encoders, generate text based on the received feature vector, and output the generated text. N1 is an integer greater than 1, and i is an integer ranging from 2 to N1. The processing module is used to adjust the input relationship of the encoder in the first neural network model to obtain the second neural network model. The Lth encoder in the second neural network model is used to receive the feature vector output by the LMth encoder. The processing module is further configured to train the second neural network model using training data to obtain a third neural network model; The processing module is also used to send the third neural network model to the target device; The acquisition module is further configured to: acquire the weight of each encoder among the N1 encoders; The processing module is further configured to: determine the encoders corresponding to the first M weights after arranging the N1 weights corresponding to the N1 encoders in ascending order as the M encoders between the Lth encoder and the LMth encoder.

8. A training device for a text generation model, characterized in that, The system includes a processor for retrieving a computer program from memory, which, when executed, performs the method according to any one of claims 1 to 4.

9. A computer-readable storage medium for storing a computer program, characterized in that, The computer program includes code for performing the method according to any one of claims 1 to 4.

Images