A training method and device for a neural network model

Abstract

The present application provides a training method and device for a neural network model, wherein the method comprises: obtaining a training sample set comprising at least one training sample pair, wherein each training sample pair comprises a training sample and an original training label; training multiple original models according to the training sample set, determining prediction labels corresponding to the training samples based on the multiple original models, and generating target training labels through the prediction labels and the original training labels; and training a target model through the training samples and the target training labels until a training stop condition is reached.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a training method and device for a neural network model, a text translation method and device based on a neural network, a computing device, and a computer-readable storage medium. Background Art

[0002] Neural network model is a basic part of solving natural language processing (NLP) tasks, and plays an important role in machine translation, reading comprehension or entity recognition. In recent years, as neural networks have been successfully applied to the construction of language models, neural network models are used to process natural language, obtain the feature representation corresponding to each character (token) in the corpus after integrating the semantic information of the whole text, and classify and predict the probability of each character in the corpus.

[0003] Most current neural network machine translation models use an end-to-end model structure, namely, an encoder and a decoder. The encoder inputs a word vector and generates an encoding vector. The decoder decodes the target input word vector and the encoding vector to generate a decoding vector, and finally generates a target translation sentence based on the decoding vector. Generally speaking, the representation ability of a model is positively correlated with the size of the model, that is, the larger the model, the stronger its representation ability. The stronger the model representation ability, the better the model performance. Therefore, how to train a neural network model based on a large amount of corpus to obtain better model performance and accuracy is an urgent problem to be solved in the current natural language processing task. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a training method and apparatus for a neural network model, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a training method for a neural network model is provided, comprising:

[0006] Acquire a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label;

[0007] Training a plurality of original models according to the training sample set, determining prediction labels corresponding to the training samples based on the plurality of original models, and generating target training labels through the prediction labels and the original training labels;

[0008] The target model is trained using the training samples and the target training labels until a training stop condition is reached.

[0009] According to a second aspect of the embodiments of this specification, a text translation method based on a neural network is provided, comprising:

[0010] Get the sentence to be translated;

[0011] Inputting the sentence to be translated into a text translation model, wherein the text translation model is trained according to the training method of the neural network model described in the present application;

[0012] The sentence to be translated is translated by the text translation model, and a target translation sentence corresponding to the sentence to be translated is output.

[0013] According to a third aspect of the embodiments of this specification, there is provided a training device for a neural network model, comprising:

[0014] A training sample acquisition module, configured to acquire a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label;

[0015] A label distillation module is configured to train a plurality of original models according to the training sample set, determine the predicted labels corresponding to the training samples based on the plurality of original models, and generate a target training label through the predicted labels and the original training labels;

[0016] The model training module is configured to train the target model using the training samples and the target training labels until a training stop condition is reached.

[0017] According to a fourth aspect of the embodiments of this specification, a text translation device based on a neural network is provided, comprising:

[0018] A sentence acquisition module is configured to acquire sentences to be translated;

[0019] A sentence input module, configured to input the sentence to be translated into a text translation model, wherein the text translation model is trained according to the training method of the neural network model described in the present application;

[0020] The text translation module is configured to translate the sentence to be translated through the text translation model and output a target translation sentence corresponding to the sentence to be translated.

[0021] According to a fifth aspect of an embodiment of this specification, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the training method of the neural network model when executing the instructions.

[0022] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer instructions, and when the instructions are executed by a processor, the steps of the training method of the neural network model are implemented.

[0023] This application is based on the theoretical foundation of model pre-training and model distillation, adopts the reverse logic of model distillation, and uses multiple pre-trained original models to assist the training of the target model. By adopting original models with multiple different structures, the problem of single structure of the original model is avoided, and the diversity of training data and models is fully reflected. The original model is then used to optimize the training data of the target model, so that the target model has the processing capability of each original model, which improves the accuracy and prediction accuracy of the target model, and greatly improves the representation ability and model performance of the entire target model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a structural block diagram of a computing device provided in an embodiment of the present application;

[0025] Figure 2 is a flow chart of a training method for a neural network model provided in an embodiment of the present application;

[0026] Figure 3 is another flow chart of the training method of the neural network model provided in the embodiment of the present application;

[0027] Figure 4 is another flow chart of the training method of the neural network model provided in the embodiment of the present application;

[0028] Figure 5 is another flow chart of the training method of the neural network model provided in the embodiment of the present application;

[0029] Figure 6 is another flow chart of the training method of the neural network model provided in the embodiment of the present application;

[0030] Figure 7 is another flow chart of the training method of the neural network model provided in the embodiment of the present application;

[0031] Figure 8 is a flowchart of a neural network-based text translation method provided in an embodiment of the present application;

[0032] Fig. 9is another flow chart of the neural network-based text translation method provided in an embodiment of the present application;

[0033] Fig.10 is another flow chart of the neural network-based text translation method provided in an embodiment of the present application;

[0034] Fig.11 It is a structural schematic diagram of a training device for a neural network model provided in an embodiment of the present application;

[0035] Fig.12 It is a schematic diagram of the structure of the machine translation device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0036] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0037] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0038] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0039] First, the terms involved in one or more embodiments of the present invention are explained.

[0040] Model distillation: Model distillation is the process of transferring the knowledge of a pre-trained model to another model with a simpler structure. The pre-trained model is usually called a complex model (Teacher Model), and the model with a simpler structure is called a simple model (Student Model). Model distillation usually involves training a complex model to guide the simple model, and using the output of the pre-trained complex model to assist the training of the simple model. In model distillation, the training of the complex model and the simple model are completed at different stages. Therefore, this model training method is also a multi-stage training method.

[0041] Soft label: also known as soft target, refers to the predicted classification probability output by the original model after generalization of the distillation temperature parameter.

[0042] Hard label: also known as hard target, refers to the original training label in the training sample set.

[0043] Distillation temperature parameter: The distillation temperature parameter (Temperature) refers to the hyperparameter in the distillation objective function. The larger the distillation temperature parameter, the smoother the probability distribution produced.

[0044] Generalization ability: In machine learning methods, generalization ability generally refers to the ability of a neural network model to predict unknown data. In actual situations, the generalization ability of a learning method is usually evaluated by test error. If the generalization ability of a model is poor without considering the lack of data, the reason is basically that the optimization of the loss function has not reached the global optimum.

[0045] In the present application, a training method and apparatus for a neural network model, a computing device, and a computer-readable storage medium are provided, which are described in detail one by one in the following embodiments.

[0046] Figure 1 The structure block diagram of a computing device 100 according to an embodiment of the present specification is shown. The components of the computing device 100 include but are not limited to a memory 110 and a processor 120. The processor 120 is connected to the memory 110 via a bus 130, and the database 150 is used to store data.

[0047] The computing device 100 also includes an access device 140 that enables the computing device 100 to communicate via one or more networks 160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0048] In one embodiment of the present specification, the above components of the computing device 100 and Figure 1 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 1 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0049] The computing device 100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 100 may also be a mobile or stationary server.

[0050] The main purpose of this application is to use the reverse logic of the model distillation principle according to specific natural language processing tasks, such as machine translation tasks, reading comprehension tasks or entity recognition tasks, and adopt multiple original models with different structures in the pre-training stage, wherein the original models include but are not limited to classification models, machine translation models, reading comprehension models or entity recognition models, and can also use specific natural language processing tasks as the main task and other related natural language processing tasks and classification tasks as auxiliary tasks, and combine the above models according to the corresponding needs of the target model.

[0051] Furthermore, the multiple pre-trained original models are used as relatively simple "simple models" in model distillation, and the target model corresponding to the natural language processing task of the main task is used as the relatively complex "complex model" in model distillation. Multiple "simple models" are used to assist the training process of the "complex model", and finally a relatively complex target model is obtained through multiple relatively simple original models.

[0052] It should be noted that the “simple” and “complex” described in this application are relative concepts within a specific range, that is, only relative to the original model and the target model. The target model is “complex” compared with the original model in terms of parameter size, structural complexity and model scale.

[0053] For example, for the Bidirectional Encoder Representations from Transformers (BERT) model, the vector representation of each character after conversion may be 12×768 dimensions or 6×516 dimensions. In this case, the 12×768-dimensional BERT model is a "complex model" and the 6×516-dimensional BERT model is a "simple model".

[0054] The processor 120 may execute Figure 2 Steps in the method shown. Figure 2 is a schematic flow chart showing a training method for a neural network model according to an embodiment of the present application, including steps 202 to 206.

[0055] Step 202: Obtain a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label.

[0056] In an embodiment of the present application, the present application first needs to obtain a sufficient amount of training data of different categories to pre-train multiple original models respectively, wherein the acquisition methods include obtaining from encyclopedia websites, obtaining from databases, and obtaining from preset knowledge graphs, etc., and the acquired training data can also come from different knowledge fields, thereby enriching the diversity of the data and improving the application field of the model.

[0057] Specifically, the training data includes training samples and original training labels, and the training samples and original training labels include both classification samples and classification labels corresponding to the classification samples related to the classification task as an auxiliary task, and training samples and corresponding original training labels associated with the natural language processing task as the main task.

[0058] For example, when the main task is a machine translation task, the training data of the original model includes Chinese and foreign language comparison sentences on the basis of the classification samples related to the classification task and the classification labels corresponding to the classification samples; when the main task is a reading comprehension task, the training data of the original model includes reading comprehension questions and answers on the basis of the classification samples related to the classification task and the classification labels corresponding to the classification samples; when the main task is an entity recognition task, the training data of the original model includes text corpus and entities contained therein on the basis of the classification samples related to the classification task and the classification labels corresponding to the classification samples.

[0059] Step 204: train multiple original models according to the training sample set, determine prediction labels corresponding to the training samples based on the multiple original models, and generate target training labels through the prediction labels and the original training labels.

[0060] In an embodiment of the present application, when pre-training the original model, for any original model, the parameters of the model can be randomly initialized, and then according to the pre-training task corresponding to the original model, each original model can be jointly trained in a supervised manner using the corresponding training data, thereby obtaining multiple original training models, and then through the reverse logic of model distillation, the final target model is designed according to the natural language processing task of the main task, and the training data of the target model is optimized and adjusted through multiple original models, thereby improving the generalization ability of the target model to treat prediction data when processing the natural language task of the main task.

[0061] Specifically, if Figure 3 As shown, multiple original models are trained according to the training sample set, including steps 302 to 306.

[0062] Step 302: According to the training samples corresponding to each original model, an original prediction result generated in response to the training samples as input is obtained.

[0063] Step 304: Calculate the loss value of each original model according to the original prediction results corresponding to the plurality of training samples and the original training labels corresponding to the plurality of training samples.

[0064] Step 306: Adjust the parameters of each original model according to the loss value of each original model until the loss value of each original model is less than a preset threshold.

[0065] In a specific embodiment of the present application, when the main task is a machine translation task, three original models are included, namely, machine translation model a, classification model b and entity recognition model c. The training data corresponding to the machine translation model a, the training data corresponding to the classification model b and the training data corresponding to the entity recognition model c are selected respectively. The machine translation model a, the classification model b and the entity recognition model c are jointly trained simultaneously, and then the loss value of each model is made less than a preset threshold by calculating the loss value and adjusting the model parameters, so as to meet the model training stop condition.

[0066] This application increases the diversity of models by using rich training data to train original models of multiple categories in the pre-training stage, which is beneficial to improving the training effect and representation ability of the target model.

[0067] In the embodiments of the present application, Figure 4 As shown, the predicted labels corresponding to the training samples are determined based on the multiple original models, and the target training labels are generated by the predicted labels and the original training labels, including steps 402 to 404.

[0068] Step 402: configuring a corresponding distillation temperature parameter for each of the original models, and determining a prediction label corresponding to each training sample based on each configured original model.

[0069] In the above embodiment, if Figure 5 As shown, step 402 specifically includes steps 502 to 504.

[0070] Step 502: Configure the corresponding distillation temperature parameters in the normalization layer of each original model.

[0071] In the process of pre-training multiple original models according to the auxiliary tasks, it is necessary to configure a corresponding distillation temperature parameter T in the normalization layer of the model structure of each original model, as shown in the following formula:

[0072]

[0073] Among them, z i represents the score of each category in the input normalization layer of the original model, q i It represents the probability of each category calculated by the normalization layer. The distillation temperature parameter T is a hyperparameter, and its range can be between 1 and 25. The larger the distillation temperature parameter T is, the smoother the probability distribution generated by the corresponding original model is.

[0074] Step 504: Input each of the training samples into each configured original model respectively, and determine the prediction label corresponding to each training sample according to the prediction result output by each configured original model.

[0075] By inputting each training sample into each configured original model for classification prediction, the normalization layer after each original model is configured with the distillation temperature parameter T can output a relatively smooth original classification probability vector corresponding to each training sample, and then select the category with the largest probability value as the prediction result of the current original model for the current training sample.

[0076] Specifically, according to the prediction results output by each configured original model, the prediction label corresponding to each training sample is determined, which specifically includes steps S5041 to S5042.

[0077] Step S5041: Obtain the original prediction probability corresponding to each training sample output by each original model.

[0078] Step S5042: Calculate the average value of the original prediction probability corresponding to each training sample output by each original model to obtain the prediction label corresponding to each training sample.

[0079] Since there are multiple original models and their model structures are different, multiple original classification probabilities corresponding to each training sample can be obtained based on the multiple original models. The average value of the multiple original classification probabilities is taken as the final probability output vector of the current training sample, and finally the prediction label corresponding to each training sample is obtained.

[0080] The present application uses the model distillation method to configure corresponding distillation temperature parameters in the normalization layer of each original model, thereby adjusting the probability distribution corresponding to the prediction results output by the original model, so that the classification probabilities between the original prediction probabilities corresponding to different classification categories output by the original model are smoother, thereby improving the accuracy and generalization of the target model.

[0081] Step 404: weight fusion is performed on the predicted label corresponding to each training sample and the original training label to obtain the target training label corresponding to each training sample.

[0082] In the above embodiment, if Figure 6 As shown, step 404 specifically includes steps 602 to 606.

[0083] Step 602: respectively configure corresponding fusion weight parameters for the predicted label and the original training label corresponding to each training sample.

[0084] In an embodiment of the present application, the predicted label corresponding to each of the training samples is defined as a soft label (soft_target), and the original training label corresponding to each of the training samples is defined as a hard label (hard_target), wherein the soft label is the cross entropy of the category probability output of the original model and the category probability output of the target model, and the hard label is the cross entropy of the category probability output of the original model and the true value of the original training label, and each soft label and hard label is configured with a fusion weight parameter a and a fusion weight parameter b for controlling the label fusion weight.

[0085] Step 604: weighted calculation is performed on the predicted label and the original training label corresponding to each training sample using the fusion weight parameter to obtain the target prediction probability corresponding to each training sample.

[0086] In an embodiment of the present application, first, the classification category to which each training sample belongs is determined based on the original training label corresponding to each training sample in the training sample collection, and then the classification probability corresponding to the classification category to which each training sample belongs is set to 1, and the classification probability corresponding to other classification categories corresponding to each training sample is set to 0, thereby obtaining a hard label corresponding to each training sample.

[0087] Furthermore, the target prediction probability of each training sample corresponding to each classification category can be calculated by the following formula:

[0088] Target=a*hard_target+b*soft_target;

[0089] Among them, the sum of the fusion weight parameter a and the fusion weight parameter b is 1.

[0090] Step 606: Generate a target training label corresponding to each training sample according to the target prediction probability corresponding to each training sample.

[0091] In an embodiment of the present application, the target prediction probability of each training sample corresponding to each classification category is aggregated to finally generate a target training label corresponding to each training sample.

[0092] This application obtains target training labels for training the target model by fusing the predicted labels with the original training labels. Since the predicted labels have higher entropy, they can provide more training information than the original training labels. The target model trained in this way has performance effects that exceed those of multiple original models, thereby enabling the target model to obtain the generalization ability of the original model and have a stronger representation ability for complex natural languages.

[0093] Step 206: Train the target model using the training samples and the target training labels until a training stop condition is reached.

[0094] In an embodiment of the present application, the present application first obtains the target prediction result generated by the target model in response to the training sample as input, and then calculates the loss value of the target model based on the target prediction result and the target training label corresponding to each training sample, and then adjusts the parameters of the target model according to the loss value of the target model until the loss value of the target model is less than a preset threshold, thereby reaching the training stop condition.

[0095] In the present application, the target model is trained by using the target training labels obtained after the final fusion and the training samples as the training data of the target model, and during the training, the distillation temperature parameter T of the normalization layer of the original model is set to be consistent with the distillation temperature parameter T of the normalization layer of the target model, and the training is performed in the manner of supervised model training, and then the loss value is calculated and the parameters of the target model are reversely adjusted until the loss value of the target model is less than the preset threshold value, and the training stop condition is reached. It should be noted that when the target model is applied, the distillation temperature parameter T of the target model is reset to 1, that is, the most original normalization layer is used to predict the specific application data.

[0096] This application is based on the theoretical foundation of model pre-training and model distillation, adopts the reverse logic of model distillation, and uses multiple pre-trained original models to assist the target model in training. By using original models with multiple different structures for pre-training, the problem of single structure of the original model is avoided, and the diversity of training data and models is fully reflected. Then, the original model is used to optimize the training data of the target model, so that the target model has the processing capability of each original model, which improves the accuracy and prediction accuracy of the target model, and greatly improves the representation ability and model performance of the entire target model.

[0097] Figure 7 A training method for a neural network model according to an embodiment of the present specification is shown. The training method for the neural network model is described by taking an original model M and a target model N as an example, and includes steps 702 to 712.

[0098] Step 702: Obtain a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label.

[0099] The step 702 is identical to the step 202, please refer to the description of the step 202, and this application will not repeat it here.

[0100] Step 704: Train t original models M using the training samples and original training labels to obtain t original models M, where t is a positive integer greater than or equal to 1.

[0101] Among them, according to the natural language processing task corresponding to the target model N as the main task, the t original models M can be designed as models with different structures, including but not limited to classification models, machine translation models, reading comprehension models or entity recognition models, etc., and the required training data of each original model is selected according to its structure and parameters for synchronous joint training to obtain t original models M of different types.

[0102] Step 706: respectively configure corresponding distillation temperature parameters T for the t original models M, input the training samples into the t original models M for classification, and obtain t original prediction probabilities corresponding to the t original models M.

[0103] Step 708: Calculate the average of t original prediction probabilities corresponding to t original models to obtain a prediction label corresponding to each of the training samples.

[0104] Step 710: weight fusion is performed on the predicted label corresponding to each training sample and the original training label to obtain the target training label corresponding to each training sample.

[0105] Step 712: Perform supervised training on the target model N using the training samples and the target training labels until a training stop condition is reached.

[0106] This application first pre-trains multiple original models as "simple models", and then, based on the idea of ​​model distillation, uses the output of the pre-trained original models to assist in training a large-scale target model, namely a "complex model". This strategy is a multi-stage training method, where one model is trained in each stage, and the pre-trained model is used to assist in the training of subsequent models, ultimately completing the training of the entire target model.

[0107] The processor 120 may also execute Figure 8 Steps in the method shown. Figure 8 80 is a schematic flowchart showing a neural network-based text translation method according to an embodiment of the present application, including steps 802 to 806.

[0108] Step 802: Obtain the sentence to be translated.

[0109] When the main task is a machine translation task, the application first obtains a translated sentence that needs to be translated, such as "I love you China".

[0110] Step 804: input the sentence to be translated into a text translation model, wherein the text translation model is trained according to the training method of the neural network model described in the present application.

[0111] Accordingly, a corresponding text translation model is designed according to the main task as the target model of the present application, and multiple original models are further pre-trained to optimize and adjust the training data of the text translation model to train a high-performance text translation model.

[0112] Specifically, if Fig. 9 As shown, the text translation model is trained in the following way:

[0113] Step 902: Pre-train multiple original models according to the translation training data, wherein the original models include a machine translation model and a classification model.

[0114] In the case where the target model is a text translation model, the translation task and the classification task can be used as auxiliary tasks of the main task, and several machine translation models and several classification task models can be designed as original models. The translation training data required for pre-training multiple original models is obtained, including the original corpus, the translation corpus corresponding to each original corpus, and the classification label corresponding to each character in the original corpus, etc. The translation task model is trained according to the multiple original corpuses and the translation corpus corresponding to each original corpus; the translation task model is trained according to the multiple original corpuses and the translation corpus corresponding to each original corpus, wherein the training data of each classification task model may be different, and the structure of each classification task model may also be different, which can be set as needed, and this application does not make specific limitations here.

[0115] It should be noted that multiple translation task models and multiple classification tasks are trained jointly, that is, multiple models are trained synchronously during training, thereby reflecting the differences between different models, and thus enabling the final text translation model to have better generalization capabilities.

[0116] Step 904: Determine the translation prediction labels corresponding to the translation training data based on the multiple original models, and generate target translation training labels through the translation prediction labels and the original translation labels corresponding to the translation training data.

[0117] Furthermore, a distillation temperature parameter T is configured for each original model, and the original translation labels in the translation training data are adjusted through multiple original models to finally obtain the target translation training labels.

[0118] Step 906: training the text translation model using the training samples corresponding to the translation training data and the target translation training labels until a training stop condition is reached.

[0119] The text translation model as the target model is trained by using the adjusted translation training data, wherein the adjusted translation training data includes a plurality of original corpora and a target translation training label corresponding to each original corpus, the plurality of original corpora are input into the text translation model, and original translation results generated by the text translation model in response to the plurality of original corpora as input are obtained, and then the loss value of the text translation model is calculated according to the original translation result corresponding to each original corpus and the target translation training label corresponding to each original corpus, and finally the parameters of the text translation model are adjusted according to the loss value of the text translation model until the loss value of the text translation model is less than a preset threshold value, so as to meet the training stop condition.

[0120] Step 806: Translate the sentence to be translated using the text translation model, and output a target translation sentence corresponding to the sentence to be translated.

[0121] In an embodiment of the present application, the word vector representation corresponding to the sentence to be translated is input into the text translation model for processing, and finally the target translation sentence corresponding to the sentence to be translated is output. For example, the sentence to be translated may be "I love you China", and the sentence to be translated "I love you China" is split into the corresponding characters "I", "love", "you" and "China" and the corresponding word vector representation is generated and input into the text translation model for processing, and finally the target translation sentence "I Love You China" corresponding to the sentence to be translated is output.

[0122] In the embodiment of the present application, step 806 specifically includes steps 1002 to 1004 .

[0123] Step 1002: Generate an encoding vector corresponding to the sentence to be translated through the encoder of the text translation model, and input the encoding vector into the decoder of the text translation model.

[0124] In the framework of the Transform model, the text translation model is mainly composed of several encoders and several decoders. Specifically, the encoder is composed of six identical encoding layers stacked together, and each encoding layer has two branches. The first branch is a multi-head self-attention mechanism (Multi-Head Attention), and the second branch is a simple fully connected feedforward network (Feed Forward). A residual connection is added outside the two branches, and then a layer normalization operation (Layer Nomalization) is performed. The decoder also stacks six decoding layers. In each decoding layer, in addition to the two branches in the encoder, the decoder also adds a third self-attention branch layer, and a residual connection is added outside the three branches, and then a layer normalization operation is performed.

[0125] Step 1004: Generate a decoding vector corresponding to the sentence to be translated through the decoder of the text translation model, and generate and output a target translation sentence corresponding to the sentence to be translated based on the decoding vector.

[0126] Each encoding layer of the encoder includes a self-attention layer and a feedforward neural network layer. The word vector representation corresponding to the sentence to be translated is processed by the self-attention layer, and then the feedforward neural network layer performs nonlinear transformation, and the input is transferred to the next encoding layer for calculation, and so on, until the last encoding layer, the last encoding layer generates the encoding vector corresponding to the sentence to be translated and maps it to the decoder. There are three operations in each decoding layer of the decoder, namely the self-attention mechanism, the encoding-decoding attention mechanism and the feedforward neural network operation, and finally the target translation sentence corresponding to the sentence to be translated is output.

[0127] This application designs a corresponding text translation model by taking the machine translation task as the main task, and designs multiple machine translation models and classification models as the original model to perform auxiliary tasks of translation and classification. By adopting a variety of original models with different structures for pre-training, in addition to the machine translation task, other natural language processing tasks are also added as auxiliary tasks, which not only enriches the training tasks but also utilizes more training data, so that the text translation model finally trained can learn more general features and improve the generalization ability of the model. Compared with the translation model trained directly, the text translation model has more superior model performance.

[0128] Corresponding to the above method embodiment, this specification also provides a training device embodiment of a neural network model. Fig.11 FIG. 1 is a schematic diagram showing a structure of a training device for a neural network model according to an embodiment of the present specification. Fig.11 As shown, the device comprises:

[0129] The training sample module 1101 is configured to obtain a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label;

[0130] The label distillation module 1102 is configured to train a plurality of original models according to the training sample set, determine the predicted labels corresponding to the training samples based on the plurality of original models, and generate target training labels through the predicted labels and the original training labels;

[0131] The model training module 1103 is configured to train the target model using the training samples and the target training labels until a training stop condition is reached.

[0132] Optionally, the label distillation module 1102

[0133] It is further configured as:

[0134] Respectively obtain a plurality of training samples in the at least one training sample pair and input them into each original model respectively;

[0135] According to the training samples corresponding to each original model, obtaining the original prediction results generated in response to the training samples as input;

[0136] Calculate the loss value of each original model based on the original prediction results corresponding to several training samples and the original training labels corresponding to several training samples;

[0137] The parameters of each original model are adjusted according to the loss value of each original model until the loss value of each original model is less than a preset threshold.

[0138] Optionally, the label distillation module 1102 includes:

[0139] A prediction label generating unit is configured to configure a corresponding distillation temperature parameter for each of the original models, and determine a prediction label corresponding to each training sample based on each configured original model;

[0140] The label fusion unit is configured to perform weighted fusion on the predicted label corresponding to each training sample and the original training label to obtain the target training label corresponding to each training sample.

[0141] Optionally, the prediction label unit is further configured as follows:

[0142] Configure the corresponding distillation temperature parameters in the normalization layer of each original model;

[0143] Each of the training samples is input into each configured original model respectively, and the prediction label corresponding to each training sample is determined according to the prediction result output by each configured original model.

[0144] Optionally, the prediction label unit is further configured as follows:

[0145] Get the original prediction probability corresponding to each training sample output by each original model;

[0146] The average value of the original prediction probability corresponding to each training sample output by each original model is calculated to obtain the prediction label corresponding to each training sample.

[0147] Optionally, the label fusion unit is further configured to:

[0148] Configure the corresponding fusion weight parameters for the predicted label and original training label corresponding to each training sample;

[0149] The predicted label and the original training label corresponding to each training sample are weightedly calculated by the fusion weight parameter to obtain the target prediction probability corresponding to each training sample;

[0150] The target training label corresponding to each training sample is generated according to the target prediction probability corresponding to each training sample.

[0151] This application is based on the theoretical foundations of model pre-training and model distillation, adopts the reverse logic of model distillation, and uses a pre-trained simple original model to assist the training of a complex target model. Through a multi-stage training mode, the training of the entire complex target model is finally completed, thereby ultimately improving the representation ability and model performance of the entire target model as well as the accuracy of the model in machine translation applications.

[0152] Corresponding to the above method embodiment, this specification also provides a text translation device embodiment based on a neural network, Fig.12 FIG. 1 is a schematic diagram showing the structure of a text translation device based on a neural network according to an embodiment of the present specification. Fig.12 As shown, the device comprises:

[0153] The sentence acquisition module 1201 is configured to acquire a sentence to be translated;

[0154] A sentence input module 1202 is configured to input the sentence to be translated into a text translation model, wherein the text translation model is trained according to the training method of the neural network model described in the present application;

[0155] The text translation module 1203 is configured to translate the sentence to be translated using the text translation model, and output a target translation sentence corresponding to the sentence to be translated.

[0156] Optionally, the text translation module 1203 is further configured to:

[0157] Generate an encoding vector corresponding to the sentence to be translated through the encoder of the machine translation model, and input the encoding vector into the decoder of the machine translation model;

[0158] A decoding vector corresponding to the sentence to be translated is generated by a decoder of the machine translation model, and a target translation sentence corresponding to the sentence to be translated is generated and output according to the decoding vector.

[0159] This application designs a corresponding text translation model by taking the machine translation task as the main task, and designs multiple machine translation models and classification models as the original model to perform auxiliary tasks of translation and classification. By adopting a variety of original models with different structures for pre-training, in addition to the machine translation task, other natural language processing tasks are also added as auxiliary tasks, which not only enriches the training tasks but also utilizes more training data, so that the text translation model finally trained can learn more general features and improve the generalization ability of the model. Compared with the translation model trained directly, the text translation model has more superior model performance.

[0160] It should be noted that the components in the device claim should be understood as the functional modules that must be established to implement the steps of the program flow or the steps of the method, and the functional modules are not actually divided or separated. The device claim defined by such a group of functional modules should be understood as the functional module architecture of the solution mainly implemented by the computer program recorded in the specification, and should not be understood as the physical device that mainly implements the solution by hardware.

[0161] An embodiment of the present application further provides a computing device, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the instructions:

[0162] Acquire a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label;

[0163] Acquire a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label;

[0164] Training a plurality of original models according to the training sample set, determining prediction labels corresponding to the training samples based on the plurality of original models, and generating target training labels through the prediction labels and the original training labels;

[0165] The target model is trained using the training samples and the target training labels until a training stop condition is reached.

[0166] An embodiment of the present application further provides a computing device, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the instructions:

[0167] Get the sentence to be translated;

[0168] Inputting the sentence to be translated into a text translation model, wherein the text translation model is trained according to the training method of the neural network model described in the present application;

[0169] The sentence to be translated is translated by the text translation model, and a target translation sentence corresponding to the sentence to be translated is output.

[0170] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the training method of the neural network model and the neural network-based text translation method as described above.

[0171] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the computer-readable storage medium and the technical scheme of the training method of the neural network model described above belong to the same concept, and the details not described in detail in the technical scheme of the computer-readable storage medium can be found in the description of the technical scheme of the training method of the neural network model described above.

[0172] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0173] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0174] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0175] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0176] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is limited only by the claims and their full scope and equivalents.

Claims

1. A training method for a neural network model, characterized in that: include: Acquire a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label; Training a plurality of original models according to the training sample set, determining prediction labels corresponding to the training samples based on the plurality of original models, and generating target training labels through the prediction labels and the original training labels, wherein the plurality of original models are determined according to the natural language processing task corresponding to the target model as the main task and have different structures, and the prediction labels are obtained by averaging a plurality of original prediction probabilities corresponding to each training sample output by the plurality of original models; generating the target training labels through the prediction labels and the original training labels, comprising: weighting the prediction labels corresponding to each training sample and the original training labels to obtain the target training labels corresponding to each training sample; The target model is trained using the training samples and the target training labels until a training stop condition is reached.

2. The method according to claim 1, characterized in that Training a plurality of original models according to the training sample set includes: Respectively obtain a plurality of training samples in the at least one training sample pair and input them into each original model respectively; According to the training samples corresponding to each original model, obtaining the original prediction results generated in response to the training samples as input; Calculate the loss value of each original model based on the original prediction results corresponding to several training samples and the original training labels corresponding to several training samples; The parameters of each original model are adjusted according to the loss value of each original model until the loss value of each original model is less than a preset threshold.

3. The method according to claim 1, characterized in that Determining the prediction labels corresponding to the training samples based on the multiple original models includes: A corresponding distillation temperature parameter is configured for each of the original models, and a prediction label corresponding to each training sample is determined based on each configured original model.

4. The method according to claim 3, characterized in that Configuring a corresponding distillation temperature parameter for each of the original models, and determining a prediction label corresponding to each training sample based on each of the configured original models, including: Configure the corresponding distillation temperature parameters in the normalization layer of each original model; Each of the training samples is input into each configured original model respectively, and the prediction label corresponding to each training sample is determined according to the prediction result output by each configured original model.

5. The method according to claim 4, characterized in that According to the prediction results output by each configured original model, the prediction label corresponding to each training sample is determined, including: Get the original prediction probability corresponding to each training sample output by each original model; The average value of the original prediction probability corresponding to each training sample output by each original model is calculated to obtain the prediction label corresponding to each training sample.

6. The method according to claim 1, characterized in that The predicted label corresponding to each training sample and the original training label are weighted and fused to obtain the target training label corresponding to each training sample, including: Configure the corresponding fusion weight parameters for the predicted label and original training label corresponding to each training sample; The predicted label and the original training label corresponding to each training sample are weightedly calculated by the fusion weight parameter to obtain the target prediction probability corresponding to each training sample; The target training label corresponding to each training sample is generated according to the target prediction probability corresponding to each training sample.

7. A text translation method based on a neural network, characterized in that: include: Get the sentence to be translated; Inputting the sentence to be translated into a text translation model, wherein the text translation model is trained according to the training method of the neural network model according to any one of claims 1 to 6; The sentence to be translated is translated by the text translation model, and a target translation sentence corresponding to the sentence to be translated is output.

8. The method according to claim 7, characterized in that The text translation model is trained in the following way: Pre-training a plurality of original models according to the translation training data, wherein the original models include a machine translation model and a classification model; Determine the translation prediction labels corresponding to the translation training data based on the multiple original models, and generate target translation training labels by using the translation prediction labels and the original translation labels corresponding to the translation training data; The text translation model is trained using the training samples corresponding to the translation training data and the target translation training labels until a training stop condition is reached.

9. The method according to claim 7, characterized in that: Translating the sentence to be translated by the text translation model and outputting a target translation sentence corresponding to the sentence to be translated includes: Generate an encoding vector corresponding to the sentence to be translated through the encoder of the text translation model, and input the encoding vector into the decoder of the text translation model; A decoding vector corresponding to the sentence to be translated is generated by a decoder of the text translation model, and a target translation sentence corresponding to the sentence to be translated is generated and output according to the decoding vector.

10. A training device for a neural network model, characterized in that: include: A training sample acquisition module, configured to acquire a training sample set including at least one training sample pair, wherein each training sample pair includes a training sample and an original training label; The label distillation module is configured to train multiple original models according to the training sample set, determine the prediction labels corresponding to the training samples based on the multiple original models, and generate target training labels through the prediction labels and the original training labels, wherein the multiple original models are determined according to the natural language processing tasks corresponding to the target models as the main tasks and have different structures, and the prediction labels are obtained by averaging multiple original prediction probabilities corresponding to each training sample output by the multiple original models; the label distillation module also includes: a label fusion unit, configured to perform weighted fusion on the prediction labels corresponding to each training sample and the original training labels to obtain the target training labels corresponding to each training sample; The model training module is configured to train the target model through the training samples and the target training labels until a training stop condition is reached.

11. The device according to claim 10, characterized in that The tag distillation module is further configured to: Respectively obtain a plurality of training samples in the at least one training sample pair and input them into each original model respectively; According to the training samples corresponding to each original model, obtaining the original prediction results generated in response to the training samples as input; Calculate the loss value of each original model based on the original prediction results corresponding to several training samples and the original training labels corresponding to several training samples; The parameters of each original model are adjusted according to the loss value of each original model until the loss value of each original model is less than a preset threshold.

12. The device according to claim 10, characterized in that The label distillation module includes: The prediction label generating unit is configured to configure a corresponding distillation temperature parameter for each of the original models, and determine a prediction label corresponding to each training sample based on each configured original model.

13. The device according to claim 12, characterized in that The predicted label generating unit is further configured to: Configure the corresponding distillation temperature parameters in the normalization layer of each original model; Each of the training samples is input into each configured original model respectively, and the prediction label corresponding to each training sample is determined according to the prediction result output by each configured original model.

14. The device according to claim 13, characterized in that The predicted label generating unit is further configured to: Get the original prediction probability corresponding to each training sample output by each original model; The average value of the original prediction probability corresponding to each training sample output by each original model is calculated to obtain the prediction label corresponding to each training sample.

15. The device according to claim 12, characterized in that The label fusion unit is further configured as follows: Configure the corresponding fusion weight parameters for the predicted label and original training label corresponding to each training sample; The predicted label and the original training label corresponding to each training sample are weightedly calculated by the fusion weight parameter to obtain the target prediction probability corresponding to each training sample; The target training label corresponding to each training sample is generated according to the target prediction probability corresponding to each training sample.

16. A text translation device based on a neural network, characterized in that: include: A sentence acquisition module is configured to acquire sentences to be translated; A sentence input module, configured to input the sentence to be translated into a text translation model, wherein the text translation model is trained according to the training method of the neural network model according to any one of claims 1 to 6; The text translation module is configured to translate the sentence to be translated through the text translation model and output a target translation sentence corresponding to the sentence to be translated.

17. The device according to claim 16, characterized in that The text translation model is trained in the following way: Pre-training a plurality of original models according to the translation training data, wherein the original models include a machine translation model and a classification model; Determine the translation prediction labels corresponding to the translation training data based on the multiple original models, and generate target translation training labels by using the translation prediction labels and the original translation labels corresponding to the translation training data; The text translation model is trained using the training samples corresponding to the translation training data and the target translation training labels until a training stop condition is reached.

18. The device according to claim 16, characterized in that The text translation module is further configured to: Generate an encoding vector corresponding to the sentence to be translated through the encoder of the text translation model, and input the encoding vector into the decoder of the text translation model; A decoding vector corresponding to the sentence to be translated is generated by a decoder of the text translation model, and a target translation sentence corresponding to the sentence to be translated is generated and output according to the decoding vector.

19. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the instructions, the steps of the method described in any one of claims 1-6 or 7-9 are implemented.

20. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by the processor, the steps of the method described in any one of claims 1-6 or 7-9 are implemented.

Images