A model training method and device, and a device for model training

Abstract

Embodiments of the present application provide a model training method and device, and a device for model training. The method comprises: obtaining training data, the training data comprising first training data unrelated to a pre-training task and second training data related to the pre-training task; inputting the first training data into a language model, pre-training the language model based on the pre-training task to obtain a candidate language model; and adjusting the candidate language model based on the second training data to obtain a target language model. Embodiments of the present application obtain a candidate language model that can be applied to the pre-training task, and the network structure, parameters, etc. are more in line with the task requirements of the pre-training task. The candidate language model is adjusted based on the second training data related to the pre-training task, the model parameters are adjusted according to the loss value of the candidate model, and the model performance is further improved, thereby improving the accuracy of the target language model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular to a model training method and device and a device for model training. BACKGROUND

[0002] At present, the pre-training language model is developing rapidly. The pre-training can obtain the pre-training language model irrelevant to specific tasks from large-scale data through self-supervised learning, and then migrate the knowledge learned by the pre-training language model from the large-scale data to the task model, and use the task model to process tasks.

[0003] However, since the pre-training language model currently used is a general-purpose model, there is no pre-training language model for specific tasks. The task model obtained by migrating the pre-training language model based on the general-purpose field has poor model performance when processing specific tasks, which leads to deviation in the processing result and low accuracy. SUMMARY

[0004] The embodiments of the present application provide a model training method, device and device for model training, which can train a language model specially applicable to pre-training tasks, and improve the model performance and accuracy.

[0005] To solve the above problems, the embodiments of the present application disclose a model training method, which comprises:

[0006] Obtaining training data, the training data comprising first training data irrelevant to pre-training tasks and second training data related to pre-training tasks;

[0007] Inputting the first training data into a language model, pre-training the language model based on the pre-training tasks to obtain a candidate language model;

[0008] Adjusting the candidate language model based on the second training data to obtain a target language model.

[0009] On the other hand, the embodiments of the present application disclose a model training device, which comprises:

[0010] A training data acquisition module is configured to acquire training data, the training data comprising first training data irrelevant to pre-training tasks and second training data related to pre-training tasks;

[0011] A pre-training module is configured to input the first training data into a language model, pre-train the language model based on the pre-training tasks to obtain a candidate language model;

[0012] a model adjustment module, configured to adjust the candidate language model based on the second training data to obtain a target language model.

[0013] In yet another aspect, an embodiment of the present application discloses a device for model training, which comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs contain instructions for executing the model training method as described in one or more of the preceding aspects.

[0014] In yet another aspect, an embodiment of the present application discloses a machine readable medium having instructions stored thereon that, when executed by one or more processors, cause an apparatus to perform the model training method as described in one or more of the preceding aspects.

[0015] An embodiment of the present application also discloses a computer program product, which comprises computer instructions stored in a computer readable storage medium and adapted to be read and executed by a processor to cause a computer device having the processor to perform the model training method as described in one or more of the preceding aspects.

[0016] Embodiments of the present application have the following advantages:

[0017] Embodiments of the present application obtain first training data irrelevant to a pre-training task and second training data related to the pre-training task, pre-train a language model based on the first training data and the pre-training task to obtain a candidate language model, and then adjust the candidate language model based on the second training data to obtain a target language model. In embodiments of the present application, the language model performs a pre-training task based on large-scale first training data in a pre-training process to obtain a candidate language model, which can be applicable to the pre-training task and has network structure and parameters more in line with the task requirements of the pre-training task. Then, the candidate language model is adjusted based on the second training data related to the pre-training task, the model parameters are adjusted according to the loss value of the candidate model, and the model performance is further improved, thereby improving the accuracy of the target language model. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 is a step flowchart of an embodiment of a model training method of the present application.

[0020] Figure 2 is an application scenario architecture diagram of a model training method of the present application;

[0021] Figure 3 is a model structure schematic diagram of the present application;

[0022] Figure 4 is a structure block diagram of a model training device embodiment of the present application;

[0023] Figure 5 is a block diagram of a device 800 for model training of the present application;

[0024] Figure 6 is a structure schematic diagram of a server in some embodiments of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0026] Method embodiments

[0027] Referring to Figure 1 , a step flowchart of a model training method embodiment of the present application is shown, which can specifically include the following steps:

[0028] Step 101, obtaining training data, the training data including first training data irrelevant to a pre-training task, and second training data related to the pre-training task.

[0029] Step 102, inputting the first training data into a language model, pre-training the language model based on the pre-training task, and obtaining a candidate language model.

[0030] Step 103, adjusting the candidate language model based on the second training data, and obtaining a target language model.

[0031] The model training method provided in the embodiments of the present application can be applied to a terminal and / or a server. Referring to Figure 2 , an application scenario architecture diagram of the model training method provided in the embodiments of the present application is shown. As Figure 2As shown, the application scenarios of the embodiments of the present application can include a terminal device 201 and a server 202. Among them, the terminal device 201 and the server 202 are connected through a wireless or wired network. The terminal device 201 includes but is not limited to a smart terminal, a computer, a personal digital assistant (PDA), a tablet computer, a vehicle-mounted device, and the like. The server 202 can be a server, a server cluster composed of several servers, or a cloud computing center. The terminal 201 and the server 202 can be used alone to execute the model training method provided in the embodiments of the present application, and the terminal 201 and the server 202 can also be used to cooperatively execute the model training method provided in the embodiments of the present application.

[0032] It should be noted that the architecture diagram in the embodiments of the present application is used to more clearly illustrate the technical solutions in the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application. For other application scenario architectures and business applications, the technical methods provided by the embodiments of the present application are also applicable to similar problems.

[0033] Among them, the pre-training task can include a text smoothing task, a speech recognition task, a text translation task, and the like. The language model is a pre-training model in a general field, for example, the language model can be any one of a BERT (Bidirectional Encoder Representations from Transformers, deep language representation model), a GPT (Generative Pre-Training, generative pre-training) model, and an ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately, efficiently learning an encoder that classifies token replacements accurately).

[0034] At present, the training process of the pre-training model usually includes two stages: a pre-training stage and a fine-tuning stage. The pre-training obtains a pre-training model irrelevant to a specific task from large-scale data through self-supervised learning; the fine-tuning stage can correct the network parameters or network structure of the pre-training model for a specific task. The pre-training model after adjustment can be used to support classification, sequence labeling, structure prediction, and sequence generation and the like. However, since the training process of the pre-training model is irrelevant to the specific task, the task model obtained based on the knowledge transfer of the pre-training model has poor processing performance for the specific task and low accuracy.

[0035] To solve this problem, the embodiment of the present application first pre-trains the language model by using the first training data irrelevant to the pre-training task, and introduces the pre-training task in the pre-training process. In this way, during the whole pre-training process, the language model performs the pre-training task based on the large-scale first training data, and the obtained candidate language model is a language model suitable for the pre-training task, and the network structure, parameters, etc. are more in line with the task requirements of the pre-training task. Then, the candidate language model is adjusted based on the second training data related to the pre-training task, the model parameters are adjusted according to the loss value of the candidate model, the model performance is further improved, and the accuracy of the target language model is improved.

[0036] In an optional embodiment of the present application, the pre-training task includes a text smoothing task, and the target language model is used for smoothing processing of text data.

[0037] It should be noted that the text smoothing task is to identify the characters in the unsmooth part of the text data and remove the unsmooth part of the text.

[0038] In the text obtained by automatic speech recognition, there are often a large amount of unsmooth content. These unsmooth content will seriously interfere with the subsequent natural language understanding task (such as syntax analysis, machine translation, etc.), because these systems are often trained on relatively smooth and standard text. The purpose of the text smoothing task is to identify the unsmooth content in the text data.

[0039] For example, the following sentences, example 1: the original sentence is "this one I in one five years of time already started to do this thing", and the smoothed sentence is "I in one five years of time already started to do this thing". Example 2: the original sentence is "I like, well, I like this thing very much", and the smoothed sentence is "I like this thing very much". Example 3: the original sentence is "this car has 4 tires, well, no, has 6 tires", and the smoothed sentence is "this car has 6 tires".

[0040] Among them, "this" and "well" in example 1 belong to the idiom and adverb in the unsmooth phenomenon, and examples 2 and 3 are examples of self-correction (Restarts) type in the unsmooth language phenomenon. The Restarts type is difficult to handle in the smoothing task, because the length of the Restarts type phrase is not fixed, the position is flexible, and even a nested structure may appear. Therefore, the research related to smoothing mainly focuses on the processing of the Restarts type.

[0041] In an optional embodiment of the present application, the step 101 of obtaining training data comprises:

[0042] Step S11, obtaining first text data irrelevant to the text smoothing task and second text data relevant to the text smoothing task;

[0043] Step S12, performing data augmentation processing on the first text data to obtain third text data;

[0044] Step S13, processing the first text data and the third text data through N pre-trained models to obtain a first score vector of the first text data and a second score vector of the third text data, the dimension value of the first score vector and the dimension value of the second score vector are both N, each dimension corresponds to an output result of a pre-trained model, wherein N is a positive integer greater than 1;

[0045] Step S14, if the first score vector and the second score vector satisfy a preset condition, determining the third text data as first training data;

[0046] Step S15, performing annotation processing on the second text data to obtain second training data.

[0047] The first text data can be a large amount of text data collected from the Internet, for example, the first text data can be text data in a paper database, or text data on a network forum, etc. The second text data is text data in the text smoothing field, for example, the second text data can be text data obtained through speech recognition technology, or text data corresponding to an interactive dialogue scene, etc. The second text data usually contains incoherent content, such as mood words "oh", "eh", "al", "wow", etc., or pause words "this", "that", etc.; or self-correcting (Restarts) type of content, for example, a word is repeated in the text: "I think everyone is more and more happy"; different content appears in the text for the same information, resulting in unclear text expression: "Last May, this May, Google began to release English transcripts", etc. The second training data can be obtained by performing annotation processing on the second text data to annotate the characters in the second text data that need to be smoothed.

[0048] Since the first text data usually does not contain incoherent parts, it is necessary to insert characters and / or punctuation marks that need to be smoothed at random positions in the first text data through a data augmentation algorithm to obtain a large amount of third text data suitable for the text smoothing task.

[0049] The third text data obtained after data augmentation does not always meet the processing requirements of the text smoothing task. To further improve the effectiveness of the training data, the first and third text data can be scored using N pre-trained models to evaluate their linguistic features. Then, based on the first score vector of the first text data and the second score vector of the third text data, the third text data is filtered, and the third text data that meets the preset conditions is used as the first training data.

[0050] It should be noted that, in this embodiment of the invention, the first text data and the third text data correspond one-to-one, therefore, the first score vector and the second score vector correspond one-to-one.

[0051] In an optional embodiment of the present invention, step S14, which states that if the first score vector and the second score vector satisfy a preset condition, then the third text data is determined as the first training data, includes:

[0052] Sub-step S141: Calculate the product of the first score vector and the second score vector to obtain the third score vector;

[0053] Sub-step S142: Perform dimensionality reduction processing on the third score vector according to preset rules to obtain an M-dimensional fourth score vector, where M is a positive integer greater than 1 and less than N;

[0054] Sub-step S143: If M is greater than or equal to 2 / N, then the third text data is determined as the first training data.

[0055] Suppose the first score vector of the first text data is (x1, x2, x3, ..., x N The second score vector of the third text data is (y1, y2, y3, ..., y). N ) Calculate the product of the first score vector and the second score vector to obtain the third score vector, which can then be represented as:

[0056] (x1*y1,x2*y2,x3*y3,…,x N *y N (1)

[0057] The third score vector is dimensionality reduced according to preset rules, which is essentially the process of reducing the dimensionality of each dimension (x) of the third score vector according to preset rules. i *y i Filter x(i∈[1,N]) to remove x(i) that do not conform to the preset rules. i *y i This results in an M-dimensional vector, where M is a positive integer greater than 1 and less than N.

[0058] The preset rule can be set according to actual needs, for example, the preset rule can be set as: x i *y i is greater than a first preset value and less than a second preset value, wherein the first preset value is less than the second preset value, and the like. The embodiments of the present application are not limited in this regard.

[0059] In an optional embodiment of the present application, the step 102 of inputting the first training data into a language model, pre-training the language model based on the pre-training task, and obtaining a candidate language model comprises:

[0060] In step S21, a first character encoding vector, a first pinyin encoding vector and a first position encoding vector of the first training data are obtained.

[0061] In step S22, the first character encoding vector, the first pinyin encoding vector and the first position encoding vector are added to obtain a first input encoding vector.

[0062] In step S23, the first input encoding vector is input into a language model, the language model is pre-trained based on at least one pre-training task including a first task, and a candidate language model is obtained, wherein the first task is used to predict characters in the first training data that need to be smoothed.

[0063] The first character encoding vector, the first pinyin encoding vector and the first position encoding vector can be obtained through feature encoding processing. Compared with the conventional language model that superimposes the character encoding vector and the position encoding vector as the input vector, the embodiment of the present application adds the pinyin encoding vector. The processing data of the text smoothing task is usually text data obtained through speech recognition. In the speech recognition process, there may be misrecognition, for example, "jian guo" is recognized as "jian guo", and the like. By adding the pinyin encoding vector, reference information is added to the language model, which can ensure that the language model can also be normally processed in the presence of misrecognition, improve the robustness of the language model, and be conducive to improving the pre-trained language model to suppress speech recognition errors.

[0064] The first task is used to predict characters in the first training data that need to be smoothed. Specifically, in the pre-training process, the language model can output a smoothing label corresponding to each character in the first training data, and the smoothing label is used to indicate whether the corresponding character should be smoothed. For example, when the smoothing label of a character is 0, it means that the character needs to be retained and does not need to be smoothed; when the smoothing label of a character is 1, it means that the character needs to be smoothed.

[0065] By performing the first task in the pre-training process, the language model can better realize the language representation related to text smoothing, and improve the processing performance of the pre-trained language model for the text smoothing task.

[0066] In an optional embodiment of the present application, the step S21 of obtaining the first character encoding vector, the first pinyin encoding vector and the first position encoding vector of the first training data comprises:

[0067] The sub-step S211 of performing the masking processing on the characters in the first training data according to the preset proportion to obtain the masked first training data;

[0068] The sub-step S212 of performing the encoding processing on the masked first training data to obtain the first character encoding vector, the first pinyin encoding vector and the first position encoding vector of the first training data.

[0069] In the embodiment of the present application, before performing the encoding processing on the first training data, the characters in the first training data can also be masked first. Specifically, the characters in the first training data with a preset proportion can be replaced by a preset character or a preset mark, for example, the characters to be masked are replaced by a "mask" mark. The preset proportion can be a relatively small value, for example, 10% or 15%.

[0070] It should be noted that the masked first training data corresponds to the pre-training task of the word masking training of the language model. In other words, the pre-training task further includes a second task for predicting the masked characters in the first training data. It can be understood that, since only the characters with the preset proportion in the masked first training data are masked, the language model can predict the masked characters based on the context of the masked characters.

[0071] In an optional embodiment of the present application, the training data of the step 103 further comprises a smoothing label of the second training data, and the step of adjusting the candidate language model based on the second training data to obtain a target language model comprises:

[0072] The step S31 of processing the second training data based on the candidate language model and a bidirectional long short-term memory network to obtain a first hidden layer vector output by the candidate language model and a second hidden layer vector output by the bidirectional long short-term memory network;

[0073] The step S32 of splicing the first hidden layer vector and the second hidden layer vector to obtain a third hidden layer vector;

[0074] Step S33, inputting the third hidden layer vector into a fully connected network layer for prediction processing to obtain a prediction result corresponding to the second training data;

[0075] Step S34, adjusting the candidate language model according to the prediction result and the smoothing label of the second training data until a loss value of the candidate language model meets a convergence condition to obtain a target language model.

[0076] Referring to Figure 3 , a model structure diagram provided by an embodiment of the present application is shown. As Figure 3 shown, the candidate language model and the Bi-directional Long Short-Term Memory (BiLSTM) are connected in parallel. The second training data is processed based on the candidate language model and the Bi-directional Long Short-Term Memory respectively to obtain a first hidden layer vector output by the candidate language model and a second hidden layer vector output by the Bi-directional Long Short-Term Memory.

[0077] Then, the first hidden layer vector and the second hidden layer vector are spliced to obtain a third hidden layer vector. The third hidden layer vector is processed by a fully connected network layer for prediction processing to obtain a prediction result corresponding to the second training data.

[0078] It should be noted that the prediction result can be a text smoothing result of the second training data. According to the prediction result and the smoothing label of the second training data, the candidate language model can be adjusted. Specifically, the model parameters of the candidate language model can be adjusted until the loss value of the candidate language model meets the convergence condition to obtain a target language model.

[0079] In an optional embodiment of the present application, the processing of the second training data based on the candidate language model and the Bi-directional Long Short-Term Memory to obtain the first hidden layer vector output by the candidate language model and the second hidden layer vector output by the Bi-directional Long Short-Term Memory in step S31 comprises:

[0080] Sub-step S311, obtaining a second character encoding vector, a second pinyin encoding vector and a second position encoding vector of the second training data;

[0081] Sub-step S312, adding the second character encoding vector, the second pinyin encoding vector and the second position encoding vector to obtain a second input encoding vector;

[0082] Sub-step S313, inputting the second input encoding vector into the candidate language model for processing to obtain the first hidden layer vector output by the candidate language model;

[0083] Sub-step S314, an acoustic feature vector corresponding to the second training data is obtained.

[0084] Sub-step S315, the acoustic feature vector is input into a bidirectional long short-term memory network for processing to obtain a second hidden layer vector output by the bidirectional long short-term memory network.

[0085] In the process of adjusting the candidate language model, the acoustic feature vector of the second training data is introduced, and the acoustic feature vector of the second training data is processed by the bidirectional long short-term memory network to obtain a second hidden layer vector.

[0086] The introduction of the acoustic feature vector can further increase the reference information of the candidate language model, and is beneficial to improve the pre-trained language model to suppress speech recognition errors.

[0087] In addition, in the embodiment of the application, in addition to introducing the acoustic feature vector of the training data in the adjustment stage, the acoustic feature vector can also be introduced in the pre-training stage. Specifically, in the pre-training process, a first acoustic feature vector corresponding to first training data can be obtained, and the first acoustic feature vector, the first character encoding vector, the first pinyin encoding vector and the first position encoding vector of the first training data are added to obtain a third input encoding vector; then, the third input encoding vector is input into a language model, and the language model is pre-trained based on at least one pre-training task including the first task to obtain a candidate language model. Finally, the candidate language model is adjusted to obtain a target language model.

[0088] In an optional embodiment of the application, the method further comprises:

[0089] Step S41, knowledge distillation is performed on the target language model to obtain a text smoothing model;

[0090] Step S42, the text smoothing model is used to perform text smoothing processing on text data obtained by speech recognition to obtain target text data.

[0091] The model structure of the target language model is usually complex, and the required calculation amount and computing resources are large, and the requirement for deployment resources is high, which is not convenient for direct deployment to services. Therefore, after obtaining the target language model, knowledge distillation can be performed on the target language model to obtain a business model with smaller parameters and relatively simple model structure, and the business model is used for specific task processing.

[0092] In summary, the embodiment of the present application obtains first training data irrelevant to a pre-training task and second training data related to the pre-training task, pre-trains a language model according to the first training data and the pre-training task to obtain a candidate language model, and then adjusts the candidate language model according to the second training data to obtain a target language model. In the embodiment of the present application, the language model performs a pre-training task based on large-scale first training data in a pre-training process to obtain a candidate language model, the candidate language model can be applicable to the pre-training task, and network structure, parameters and the like are more in line with task requirements of the pre-training task. Then, the candidate language model is adjusted based on second training data related to the pre-training task, model parameters are adjusted according to a loss value of the candidate model, and model performance is further improved, thereby improving accuracy of the target language model.

[0093] It should be noted that, for the method embodiment, in order to simply describe, it is expressed as a series of action combinations, but those skilled in the art should know that the embodiment of the present application is not limited by the described action sequence, because according to the embodiment of the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiment of the present application.

[0094] Device embodiment

[0095] Reference Figure 4 , a structural block diagram of a model training device embodiment of the present application is shown, the device can include:

[0096] The training data acquisition module 401 is configured to acquire training data, the training data including first training data irrelevant to a pre-training task and second training data related to the pre-training task.

[0097] The pre-training module 402 is configured to input the first training data into a language model, pre-train the language model based on the pre-training task to obtain a candidate language model.

[0098] The model adjustment module 403 is configured to adjust the candidate language model based on the second training data to obtain a target language model.

[0099] Optionally, the pre-training task includes a text smoothing task, and the target language model is configured to perform smoothing processing on text data.

[0100] Optionally, the pre-training module includes:

[0101] a first encoding vector obtaining submodule, configured to obtain a first character encoding vector, a first pinyin encoding vector and a first position encoding vector of the first training data;

[0102] a first input encoding vector determining submodule, configured to add the first character encoding vector, the first pinyin encoding vector and the first position encoding vector to obtain a first input encoding vector;

[0103] a pre-training submodule, configured to input the first input encoding vector into a language model, pre-train the language model based on at least one pre-training task including a first task, to obtain a candidate language model, the first task being used to predict a character in the first training data that needs to be smoothed.

[0104] Optionally, the first encoding vector obtaining submodule comprises:

[0105] a masking processing unit, configured to mask characters in the first training data according to a preset proportion to obtain masked first training data;

[0106] an encoding processing unit, configured to perform encoding processing on the masked first training data to obtain the first character encoding vector, the first pinyin encoding vector and the first position encoding vector of the first training data.

[0107] Optionally, the pre-training task further comprises a second task, the second task being used to predict a masked character in the first training data.

[0108] Optionally, the training data further comprises a smoothing label of the second training data, and the model adjusting module comprises:

[0109] a hidden layer vector obtaining submodule, configured to process the second training data based on the candidate language model and a bidirectional long short-term memory network to obtain a first hidden layer vector output by the candidate language model and a second hidden layer vector output by the bidirectional long short-term memory network;

[0110] a vector splicing submodule, configured to splice the first hidden layer vector and the second hidden layer vector to obtain a third hidden layer vector;

[0111] a prediction processing submodule, configured to input the third hidden layer vector into a fully connected network layer to perform prediction processing to obtain a prediction result corresponding to the second training data;

[0112] a model adjusting submodule, configured to adjust the candidate language model according to the prediction result and the smoothing label of the second training data until a loss value of the candidate language model meets a convergence condition to obtain a target language model.

[0113] Optionally, the hidden layer vector obtaining submodule comprises:

[0114] a second encoding vector obtaining unit, configured to obtain a second character encoding vector, a second pinyin encoding vector and a second position encoding vector of the second training data;

[0115] a second encoding input vector determining unit, configured to add the second character encoding vector, the second pinyin encoding vector and the second position encoding vector to obtain a second input encoding vector;

[0116] a first hidden layer vector determining unit, configured to input the second input encoding vector into the candidate language model for processing to obtain a first hidden layer vector output by the candidate language model;

[0117] an acoustic feature vector obtaining unit, configured to obtain an acoustic feature vector corresponding to the second training data;

[0118] a second hidden layer vector determining unit, configured to input the acoustic feature vector into a bidirectional long short-term memory network for processing to obtain a second hidden layer vector output by the bidirectional long short-term memory network.

[0119] Optionally, the training data obtaining module comprises:

[0120] a text data obtaining submodule, configured to obtain first text data irrelevant to a text smoothing task and obtain second text data related to the text smoothing task;

[0121] a data augmentation submodule, configured to perform data augmentation processing on the first text data to obtain third text data;

[0122] a score vector determining submodule, configured to process the first text data and the third text data by N pre-training models to obtain a first score vector of the first text data and a second score vector of the third text data, a dimension value of the first score vector and a dimension value of the second score vector are both N, each dimension corresponds to an output result of a pre-training model, wherein N is a positive integer greater than 1;

[0123] a first training data determining submodule, configured to determine the third text data as first training data if the first score vector and the second score vector satisfy a preset condition;

[0124] a second training data determining submodule, configured to perform labeling processing on the second text data to obtain second training data.

[0125] Optionally, the first training data determining submodule comprises:

[0126] a score vector calculation unit, configured to calculate a product of the first score vector and the second score vector, to obtain a third score vector;

[0127] a dimension reduction processing unit, configured to perform dimension reduction processing on the third score vector according to a preset rule, to obtain a fourth score vector with M dimensions, where M is a positive integer greater than 1 and less than N;

[0128] a first training data determination unit, configured to determine the third text data as first training data if M is greater than or equal to 2 / N.

[0129] Optionally, the apparatus further comprises:

[0130] a knowledge distillation module, configured to perform knowledge distillation on the target language model, to obtain a text smoothing model;

[0131] a text smoothing processing module, configured to perform text smoothing processing on text data obtained through speech recognition by using the text smoothing model, to obtain target text data.

[0132] To sum up, the embodiment of the present application obtains first training data irrelevant to a pre-training task and second training data related to the pre-training task, pre-trains a language model according to the first training data and the pre-training task, to obtain a candidate language model, and then adjusts the candidate language model according to the second training data, to obtain a target language model. In the embodiment of the present application, the language model performs a pre-training task based on large-scale first training data in a pre-training process, to obtain a candidate language model, which can be applicable to the pre-training task and has network structure and parameters more in line with the task requirements of the pre-training task. Then, the candidate language model is adjusted according to second training data related to the pre-training task, model parameters are adjusted according to the loss value of the candidate model, and the model performance is further improved, thereby improving the accuracy of the target language model.

[0133] For the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0134] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other.

[0135] As for the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0136] Embodiments of the present application provide a device for model training, the device comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs comprising instructions for performing the following operations:

[0137] A11, obtaining training data, the training data comprising first training data irrelevant to a pre-training task and second training data related to the pre-training task;

[0138] A12, inputting the first training data into a language model, pre-training the language model based on the pre-training task to obtain a candidate language model;

[0139] A13, adjusting the candidate language model based on the second training data to obtain a target language model.

[0140] Optionally, the pre-training task comprises a text smoothing task, and the target language model is used for smoothing processing of text data.

[0141] Optionally, the inputting the first training data into a language model, pre-training the language model based on the pre-training task to obtain a candidate language model comprises:

[0142] obtaining a first character encoding vector, a first pinyin encoding vector and a first position encoding vector of the first training data;

[0143] adding the first character encoding vector, the first pinyin encoding vector and the first position encoding vector to obtain a first input encoding vector;

[0144] inputting the first input encoding vector into a language model, pre-training the language model based on at least one pre-training task including a first task to obtain a candidate language model, the first task being used for predicting characters in the first training data that need to be smoothed.

[0145] Optionally, the obtaining a first character encoding vector, a first pinyin encoding vector and a first position encoding vector of the first training data comprises:

[0146] masking characters in the first training data according to a preset proportion to obtain masked first training data;

[0147] encoding the masked first training data to obtain a first character encoding vector, a first pinyin encoding vector and a first position encoding vector of the first training data.

[0148] Optionally, the pre-training task further comprises a second task for predicting a masked character in the first training data.

[0149] Optionally, the training data further comprises a smoothing label of the second training data, and the adjusting the candidate language model based on the second training data to obtain a target language model comprises:

[0150] processing the second training data based on the candidate language model and a bidirectional long short-term memory network to obtain a first hidden layer vector output by the candidate language model and a second hidden layer vector output by the bidirectional long short-term memory network;

[0151] splicing the first hidden layer vector and the second hidden layer vector to obtain a third hidden layer vector;

[0152] inputting the third hidden layer vector into a fully connected network layer for prediction processing to obtain a prediction result corresponding to the second training data;

[0153] adjusting the candidate language model according to the prediction result and the smoothing label of the second training data until a loss value of the candidate language model meets a convergence condition to obtain a target language model.

[0154] Optionally, the processing the second training data based on the candidate language model and a bidirectional long short-term memory network to obtain a first hidden layer vector output by the candidate language model and a second hidden layer vector output by the bidirectional long short-term memory network comprises:

[0155] obtaining a second character encoding vector, a second pinyin encoding vector and a second position encoding vector of the second training data;

[0156] adding the second character encoding vector, the second pinyin encoding vector and the second position encoding vector to obtain a second input encoding vector;

[0157] inputting the second input encoding vector into the candidate language model for processing to obtain the first hidden layer vector output by the candidate language model;

[0158] obtaining an acoustic feature vector corresponding to the second training data;

[0159] inputting the acoustic feature vector into a bidirectional long short-term memory network for processing to obtain the second hidden layer vector output by the bidirectional long short-term memory network.

[0160] Optionally, the obtaining training data comprises:

[0161] obtain first text data irrelevant to the text smoothing task, and obtain second text data relevant to the text smoothing task;

[0162] perform data augmentation processing on the first text data to obtain third text data;

[0163] perform processing on the first text data and the third text data through N pre-trained models to obtain a first score vector of the first text data and a second score vector of the third text data, a dimension value of the first score vector and a dimension value of the second score vector are both N, each dimension corresponds to an output result of a pre-trained model, wherein N is a positive integer greater than 1;

[0164] if the first score vector and the second score vector satisfy a preset condition, determine the third text data as first training data;

[0165] perform annotation processing on the second text data to obtain second training data.

[0166] Optionally, if the first score vector and the second score vector satisfy a preset condition, determining the third text data as first training data comprises:

[0167] calculate the product of the first score vector and the second score vector to obtain a third score vector;

[0168] perform dimension reduction processing on the third score vector according to a preset rule to obtain a fourth score vector of M dimensions, wherein M is a positive integer greater than 1 and less than N;

[0169] if M is greater than or equal to 2 / N, determine the third text data as first training data.

[0170] Optionally, the apparatus further configured to execute the one or more programs by one or more processors includes instructions for:

[0171] perform knowledge distillation on the target language model to obtain a text smoothing model;

[0172] perform text smoothing processing on text data obtained through speech recognition using the text smoothing model to obtain target text data.

[0173] Figure 5 is a block diagram of an apparatus 800 for model training according to an exemplary embodiment. For example, the apparatus 800 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0174] Referring toFigure 5 The device 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0175] The processing component 802 usually controls overall operations of the device 800, such as operations associated with displaying, making phone calls, data communications, camera operations and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the above methods. In addition, the processing component 802 can include one or more modules to facilitate

[0176] The memory 804 is configured to store various types of data to support operations of the device 800. Examples of these data include instructions for any application or method operating on the device 800, contact data, phonebook data, messages, pictures, videos, and so on. The memory 804 can be realized by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic or optical disk.

[0177] The power supply component 806 provides power for various components of the device 800. The power supply component 806 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing power for the device 800.

[0178] The multimedia component 808 includes a screen providing an output interface between the device 800 and a user. In some embodiments, the screen includes a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0179] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) configured to receive an external audio signal when the device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0180] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, such as a keypad, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0181] The sensor component 814 includes one or more sensors to provide various state assessments for the device 800. For example, the sensor component 814 can detect an open / closed state of the device 800, relative positioning of components, such as a display and a keypad of the device 800, a change in position of the device 800 or a component of the device 800, presence or absence of user contact with the device 800, a change in orientation of the device 800 or acceleration / deceleration of the device 800, and a temperature change of the device 800. The sensor component 814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 814 can further include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0182] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0183] In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic elements, for performing the above-described methods.

[0184] In an exemplary embodiment, a non-transitory computer readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the device 800 to complete the above-described methods. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0185] Figure 6 is a schematic diagram of a structure of a server in some embodiments of the present application. The server 1900 can have a large difference due to different configurations or performances, and can include one or more central processing units (CPUs) 1922 (for example, one or more processors) and a memory 1932, one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be temporary storage or persistent storage. The programs stored in the storage medium 1930 can include one or more modules (not shown in the figure), each of which can include a series of instruction operations in the server. Further, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.

[0186] The server 1900 can also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input / output interfaces 1958, one or more keyboards 1956, and / or one or more operating systems 1941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0187] A non-transitory computer readable storage medium, when instructions in the storage medium are executed by a processor of an apparatus (a server or a terminal), enable the apparatus to perform Figure 1 the model training method shown.

[0188] A non-transitory computer readable storage medium, when instructions in the storage medium are executed by a processor of an apparatus (a server or a terminal), enable the apparatus to perform the foregoing Figure 1 corresponding embodiment of the model training method, and therefore, will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiments related to the present application, please refer to the description of the method embodiments of the present application.

[0189] In addition, it should be noted that: the embodiments of the present application also provide a computer program product or computer program, the computer program product or computer program can include computer instructions, the computer instructions can be stored in a computer readable storage medium. The processor of the computer equipment reads the computer instructions from the computer readable storage medium, and the processor can execute the computer instructions, so that the computer equipment executes the foregoing Figure 1 corresponding embodiment of the model training method, and therefore, will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiments related to the present application, please refer to the description of the method embodiments of the present application.

[0190] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary only. The true scope and spirit of the application is indicated by the following claims.

[0191] It should be understood that the application is not limited to the precise construction which has been described above and illustrated in the accompanying drawings and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should be determined only by the appended claims.

[0192] The above description is merely that of the preferred embodiments of the application and is not intended to limit the application thereto as modifications and alterations will be readily apparent to those skilled in the art without departing from the spirit and scope of the application.

[0193] The model training method, the model training device and the device for model training provided by the application are described in detail above, the principles and implementation manners of the application are described by applying specific examples, the above embodiment descriptions are only used to help understand the method of the application and its core idea; meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation manners and application ranges will have changes, and the above descriptions should not be understood as limiting the application.

Claims

1. A model training method, characterized in that, The method comprises: obtaining training data, the training data comprising first training data unrelated to a pre-training task and second training data related to the pre-training task; inputting first character encoding vectors, first pinyin encoding vectors, first position encoding vectors and first acoustic feature vectors of the first training data into a language model, pre-training the language model based on the pre-training task to obtain a candidate language model, the pre-training task comprising a text smoothing task, a first task and a second task, the text smoothing task being used to identify characters of unfluent parts in text data and remove the unfluent parts in the text, the first task being used to predict characters in the first training data that need to be smoothed, and the second task being used to predict characters in the first training data that are masked; adjusting model parameters of the candidate language model based on smoothing labels and prediction results of the second training data to obtain a target language model; and the prediction results are text smoothing results of the second training data.

2. The method of claim 1, wherein, The pre-training task comprises a text smoothing task, and the target language model is used for smoothing processing of text data.

3. The method of claim 2, wherein, The method comprises: obtaining first character encoding vectors, first pinyin encoding vectors and first position encoding vectors of the first training data; adding the first character encoding vectors, the first pinyin encoding vectors and the first position encoding vectors to obtain a first input encoding vector; inputting the first input encoding vector into a language model, pre-training the language model based on at least one pre-training task including the first task to obtain a candidate language model.

4. The method of claim 3, wherein, The method comprises: masking characters in the first training data according to a preset proportion to obtain masked first training data; encoding the masked first training data to obtain first character encoding vectors, first pinyin encoding vectors and first position encoding vectors of the first training data.

5. The method of claim 2, wherein, The training data further comprises smoothing labels of the second training data, and the method comprises: processing the second training data based on the candidate language model and a bidirectional long short-term memory network to obtain a first hidden layer vector output by the candidate language model and a second hidden layer vector output by the bidirectional long short-term memory network; splicing the first hidden layer vector and the second hidden layer vector to obtain a third hidden layer vector; inputting the third hidden layer vector into a fully connected network layer for prediction processing to obtain a prediction result corresponding to the second training data; and According to the prediction result and the smoothing label of the second training data, the candidate language model is adjusted until a loss value of the candidate language model meets a convergence condition, and a target language model is obtained.

6. The method of claim 5, wherein, The processing of the second training data based on the candidate language model and the bidirectional long short memory network includes: Obtaining a second character encoding vector, a second pinyin encoding vector and a second position encoding vector of the second training data; Adding the second character encoding vector, the second pinyin encoding vector and the second position encoding vector to obtain a second input encoding vector; Inputting the second input encoding vector into the candidate language model for processing to obtain a first hidden layer vector output by the candidate language model; Obtaining an acoustic feature vector corresponding to the second training data; Inputting the acoustic feature vector into the bidirectional long short memory network for processing to obtain a second hidden layer vector output by the bidirectional long short memory network.

7. The method of claim 2, wherein, The method further includes: Obtaining first text data irrelevant to the text smoothing task and second text data related to the text smoothing task; Performing data augmentation processing on the first text data to obtain third text data; Processing the first text data and the third text data through N pre-training models to obtain a first score vector of the first text data and a second score vector of the third text data, the dimension value of the first score vector and the dimension value of the second score vector are both N, each dimension corresponds to an output result of a pre-training model, wherein N is a positive integer greater than 1; If the first score vector and the second score vector meet a preset condition, the third text data is determined as the first training data; Performing labeling processing on the second text data to obtain second training data.

8. The method of claim 7, wherein, If the first score vector and the second score vector meet a preset condition, the third text data is determined as the first training data, including: Calculating the product of the first score vector and the second score vector to obtain a third score vector; Performing dimension reduction processing on the third score vector according to a preset rule to obtain a fourth score vector of M dimensions, wherein M is a positive integer greater than 1 and less than N; If M is greater than or equal to 2 / N, the third text data is determined as the first training data.

9. The method of claim 2, wherein, The method further includes: Performing knowledge distillation on the target language model to obtain a text smoothing model; Performing text smoothing processing on text data obtained through speech recognition by using the text smoothing model to obtain target text data.

10. A model training apparatus, comprising: The device includes: A training data obtaining module configured to obtain training data, the training data including first training data irrelevant to a pre-training task and second training data related to the pre-training task; The pre-training module is configured to input a first character encoding vector, a first pinyin encoding vector, a first position encoding vector, and a first acoustic feature vector of the first training data into a language model, pre-train the language model based on a pre-training task to obtain a candidate language model, and the pre-training task includes a text smoothing task, a first task, and a second task, the text smoothing task is configured to identify characters of an unsmooth part in text data and remove the unsmooth part in the text, the first task is configured to predict characters in the first training data that need to be smoothed, and the second task is configured to predict characters in the first training data that are masked. The model adjustment module is configured to adjust model parameters of the candidate language model based on a smoothing label and a prediction result of the second training data to obtain a target language model, and the prediction result is a text smoothing result of the second training data.

11. The apparatus of claim 10, wherein, The pre-training task includes a text smoothing task, and the target language model is configured to perform smoothing processing on text data.

12. The apparatus of claim 11, wherein, The training data further includes a smoothing label of the second training data, and the model adjustment module is configured to process the second training data based on the candidate language model and a bidirectional long short-term memory network to obtain a first hidden layer vector output by the candidate language model and a second hidden layer vector output by the bidirectional long short-term memory network. The first hidden layer vector and the second hidden layer vector are spliced to obtain a third hidden layer vector. The third hidden layer vector is input into a fully connected network layer for prediction processing to obtain a prediction result corresponding to the second training data. The candidate language model is adjusted according to the prediction result and the smoothing label of the second training data until a loss value of the candidate language model meets a convergence condition to obtain a target language model.

13. The apparatus of claim 11, wherein, The device further includes: The first determination module is configured to perform knowledge distillation on the target language model to obtain a text smoothing model. The second determination module is configured to perform text smoothing processing on text data obtained by speech recognition by using the text smoothing model to obtain target text data.

14. An apparatus for model training, comprising: The device includes a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for executing the model training method of any one of claims 1 to 9.

15. A machine-readable medium having instructions stored thereon that, when executed by one or more processors, cause an apparatus to perform the model training method of any one of claims 1 to 9.

16. A computer program product, characterised in that, The computer program product includes computer instructions stored in a computer readable storage medium and adapted to be read and executed by a processor to cause a computer device with the processor to perform the model training method of any one of claims 1 to 9.

Images