Model training method, device and computer-readable storage medium

Abstract

To reduce an influence caused by insufficient covering of a label word, and improve the performance of a preliminary training language model.SOLUTION: A model training method includes: a step 11 of acquiring a plurality of label words in different labels, and calculating the average vector of the label words for each of the labels to obtain a word average vector; a step 12 of inserting a masking text including a prediction word to be covered by masking into an input text to constitute a template for creating a prediction text; and a step 13 of training a preliminary training language model by using a first training sample, generating a word vector of the prediction word in a first prediction text by using the preliminary training language model, determining a prediction label of the first training sample based on the similarity between the word vector of the prediction word and the word average vector for each of the labels, and optimizing the preliminary training language model based on the prediction label of the first training sample and a labeled label.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to machine learning and natural language processing (NLP), and in particular to a model training method, apparatus and computer-readable storage medium. [Background technology]

[0002] A pre-trained model is a network that has been trained and saved with a large amount of data. It can be understood as a model created to solve a similar problem. When a new problem occurs, there is no need to train a new model from scratch. Instead, you can start directly from this model and solve the new problem with a simple learning process through fine-tuning the model.

[0003] The pre-trained model is an application of migration learning. When a neural network uses data to learn a model, it obtains information from the data, which is essentially the weights of one multi-layer network. When the weights are extracted and moved to another network, the other network learns the characteristics of that network and the knowledge it possesses. In the field of natural language processing, the words to be processed are converted into vectors using the word embedding method, and the word vectors are trained using a method based on the Transformers framework to obtain a pre-trained model, which can then be fine-tuned, that is, modified for a specific task. In the field of natural language processing, the pre-trained model is also called the pre-trained language model (PLM).

[0004] Tuning a pre-trained language model by constructing task-specific prompts is a promising approach (prompt-tuning) in text classification tasks. In text classification tasks, constructing a thesaurus can transform the classification problem into a problem of predicting and labeling related words. Thus, for such problems, we create a template containing a mask [MASK] and have the pre-trained language model predict the word in the position of [MASK]. This turns the classification task into a masking language modeling problem.

[0005] The core idea of ​​prompt-tuning is to insert text fragments (templates) into the input, turning the classification problem into a masked language modeling problem. The key to this idea is to use a language representator, i.e., to build a mapping between the label word space and the label space. Currently, language representators are usually constructed manually or searched by gradient descent, and the number of label words they contain is usually limited, so they may not cover all the label words for each label. Therefore, the limited coverage leads to large bias and high variance in the prediction results. Summary of the Invention [Problem to be solved by the invention]

[0006] The technical problem to be solved by the embodiments of the present invention is to provide a model training method, an apparatus, and a computer-readable storage medium, which map more samples to labels, reduce the impact of insufficient coverage of label words, and improve the performance of a pre-trained language model. [Means for solving the problem]

[0007] According to one embodiment of the present invention, at least one example includes a step of obtaining a plurality of labeled words in different labels, calculating an average vector of the labeled words for each label to obtain a word average vector, a step of constructing a template, the template being used to generate a predicted text by inserting a masking text into an input text, the masking text including a predicted word to be covered by the masking, and a step of training a pre-training language model using a first training set including a plurality of labeled first training samples, the step of training the pre-training language model including: generating a first predicted text based on the template using the first training sample as an input text; generating a word vector of a predicted word in the first predicted text using the pre-training language model, and determining a predicted label of the first training sample based on a similarity between the word vector of the predicted word and the word average vector of each label; and optimizing the pre-training language model based on the predicted label of the first training sample and the labeled label.

[0008] Also, according to at least one embodiment of the present invention, determining the predicted label of the first training sample based on the similarity between the word vector of the predicted word and the word mean vector of each label includes calculating a distance between the word vector of the predicted word and the word mean vector of each label, selecting a minimum distance, and setting the label corresponding to the minimum distance as the predicted label of the first training sample.

[0009] According to at least one embodiment of the present invention, the method further includes the steps of: obtaining unlabeled second training samples of a plurality of labels after training a pre-training language model using the first training set; determining predicted labels of the second training samples using the pre-training language model; and labeling the second training samples based on the determined predicted labels of the second training samples to generate a second training set.

[0010] Also, according to at least one embodiment of the present invention, the step of constructing a plurality of the templates, generating different predictive texts according to different templates, training one pre-training language model for each of the templates using the first training set to obtain a plurality of pre-training language models, and determining a predicted label of the second training sample using the pre-training language models includes: determining a probability for each label of the second training sample using each pre-training language model; and setting the label with the maximum probability as the predicted label of the second training sample.

[0011] According to at least one embodiment of the present invention, the steps of constructing a plurality of the templates, generating different predicted texts according to different templates, training one pre-training language model for each of the templates using the first training set to obtain a plurality of pre-training language models, and determining a predicted label of the second training sample using the pre-training language models include: measuring an accuracy index of label prediction for each pre-training language model using a test set, and setting a weight that is positively correlated with the accuracy index for each pre-training language model based on the accuracy index; determining a probability for each label of the second training sample using each pre-training language model; performing a weighted sum of the probabilities for the label of the second training sample determined for each pre-training language model based on the weight of the pre-training language model for each label, and calculating a probability statistic for the label of the second training sample; and setting the label with the largest probability statistic as the predicted label of the second training sample.

[0012] Also, in accordance with at least one embodiment of the present invention, the method further includes, after generating the second training corpus, training a final pre-training language model using the second training corpus.

[0013] According to another aspect of the present invention, at least one embodiment provides a model training apparatus including: a calculation module that obtains a plurality of labeled words at different labels, and calculates an average vector of the labeled words for each label to obtain a word average vector; a construction module that constructs a template, the template being used to generate a predicted text by inserting a masking text into an input text, the masking text including a predicted word to be covered by the masking; and a first training module that trains a pre-trained language model using a first training set including a plurality of labeled first training samples, the first training module generating a first predicted text based on the template using the first training sample as an input text, generating word vectors of predicted words in the first predicted text using the pre-trained language model, determining a predicted label of the first training sample based on a similarity between the word vector of the predicted word and the word average vector of each label, and optimizing the pre-trained language model based on the predicted label of the first training sample and the labeled label.

[0014] Furthermore, according to at least one embodiment of the present invention, when determining the predicted label of the first training sample based on the similarity between the word vector of the predicted word and the word mean vector of each label, the first training module calculates a distance between the word vector of the predicted word and the word mean vector of each label, selects the minimum distance, and sets the label corresponding to the minimum distance as the predicted label of the first training sample.

[0015] Also, according to at least one embodiment of the present invention, the method further includes a first acquisition module for acquiring a plurality of unlabeled second training samples of labels, and a sample generation module for determining predicted labels of the second training samples using the pre-training language model, labeling the second training samples based on the determined predicted labels of the second training samples, and generating a second training set.

[0016] Also, according to at least one embodiment of the present invention, the construction module constructs a plurality of the templates and generates different predictive texts according to different templates; the first training module trains one pre-training language model for each of the templates using the first training set to obtain a plurality of pre-training language models; and the sample generation module includes a first label generation sub-module that determines a probability for each label of a second training sample using each pre-training language model and sets the label with the maximum probability as the predicted label of the second training sample.

[0017] According to at least one embodiment of the present invention, the construction module constructs a plurality of the templates and generates different predicted texts according to different templates; the first training module trains one pre-trained language model using the first training set for each of the templates to obtain a plurality of pre-trained language models; the sample generation module includes a second label generation submodule that measures an accuracy index of label prediction for each pre-trained language model using a test set, sets a weight for each pre-trained language model that is positively correlated with the accuracy index based on the accuracy index, determines a probability of each label of a second training sample using each pre-trained language model, weights and adds the probabilities of the labels of the second training sample determined for each pre-trained language model based on the weight of the pre-trained language model for each label, calculates a probability statistic for the labels of the second training sample, and sets the label with the largest probability statistic value as the predicted label of the second training sample.

[0018] Also, in accordance with at least one embodiment of the present invention, the system further includes a second training module that uses the second training corpus to train a final pre-training language model.

[0019] Furthermore, the present invention provides a computer-readable storage medium having a computer program stored therein, the computer program being executed by a processor to realize the steps of the above-mentioned model training method. Effect of the Invention

[0020] Compared with the prior art, the model training method, device and computer-readable storage medium provided in the embodiments of the present invention can identify the label of the predicted word based on the similarity between the predicted word and the word average vector of the label word in each label, so that more samples can be mapped to the label, the influence of insufficient coverage of the label word can be reduced, and the performance of the pre-trained language model can be further improved.In addition, according to the embodiments of the present invention, the second training set with more samples can be generated by labeling, so that the training set can be expanded to train the model, and the small sample problem can be better solved. [Brief description of the drawings]

[0021] In order to more clearly explain the embodiments of the present invention, the drawings used below will be briefly described. It is clear that the drawings described below are only some embodiments of the present invention, and it is easy for a person skilled in the art to obtain other drawings from these drawings without paying creative labor.

[0022] [Figure 1] FIG. 1 is a flow diagram illustrating a method for training a model according to an embodiment of the present invention.

[0023] [Diagram 2] FIG. 2 is a diagram showing the structure of a model training device according to an embodiment of the present invention.

[0024] [Diagram 3] FIG. 3 is a diagram showing another configuration of a model training apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] In order to clarify the present invention, the technical problem that the present invention is intended to solve, and the advantages of the present invention, the embodiments will be described below with reference to the drawings. In the following description, specific details such as specific configurations and elements are provided, but only to help understand the embodiments of the present invention. Therefore, it is obvious to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. In addition, for the sake of clarity and conciseness, descriptions of known functions and structures are omitted.

[0026] References throughout the specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with an embodiment is included in at least one embodiment of the present invention. Thus, references throughout the specification to "in one embodiment" or "in one embodiment" do not necessarily refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments, as appropriate.

[0027] In each embodiment of the present invention, the numbers of each step described below do not mean the order of execution. The order of execution of each step should be determined according to its function and inherent logic, and does not limit the steps of the embodiments of the present invention in any way.

[0028] Currently, the number of label words included in the language representation is relatively limited, which may lead to significant deviation and high variance in the prediction results. In order to avoid the above problem caused by the scarcity of label words in the label word space, the embodiment of the present invention provides a model training method to map more samples to labels, thereby reducing the adverse effect of insufficient coverage of label words and improving the performance of the pre-trained language model.

[0029] As shown in Figure 1, an embodiment of the present invention provides a model training method, which includes the following steps:

[0030] In step 11, a plurality of label words for different labels are obtained, and the average vector of the label words for each label is calculated to obtain a word average vector.

[0031] Here, the embodiment of the present invention constructs a linguistic representation in which multiple labels and multiple label words in each label are defined. The label words in all the labels constitute a label word space, and all the labels constitute a label space. That is, the linguistic representation establishes a mapping relationship between the label words in the label word space and the labels in the label space, which is an injective mapping relationship between the label words in the label word space and the labels in the label space, that is, for a label word, there is only one label corresponding to the label space.

[0032] Depending on the specific application scenario of the text classification task, multiple corresponding labels are set. Then, a mapping relationship between the labels and the label words is formed by setting a word containing the meaning of a certain label as the label word of that label. For example, in the application scenario of classifying emotions in movie reviews, there are usually two types of movie review labels: positive and negative. Of these, positive labels include exciting, good, fantastic, interesting..., while negative labels include bad, terrible, boring..., and there are multiple label words for different labels, as shown in Table 1.

[0033] [Table 1] In an embodiment of the present invention, in order to mitigate the influence of the label words in the label word space not covering all the label words in the label, for each label, an average vector of all the label words in the label is calculated as the word average vector of the label, and is used for subsequent label mapping.

[0034] In addition, in order to maximize the number of label words, the embodiment of the present invention uses an external knowledge database to expand the label word space of the linguistic expresser and increase the label words in each label. The structure of the knowledge database has a range relationship. In the geometric structure of the knowledge database, related entities are connected by edges, and unrelated entities are either related or unrelated through multiple hops. Therefore, by fusing the information in the knowledge database, multiple words can be expanded for one classification.

[0035] In step 12, a template is constructed for generating predictive text by inserting masked text, including predicted words covered by masking, into the input text.

[0036] Here, the template inserts masking text into the input text to generate predicted text, turning the classification problem into a masking language modeling problem. Specifically, a template is constructed from the input text and the text classification result. For example, to classify a sentence, the following template is constructed:

[0037] A[MASK]question:x. In it, [MASK] represents the predicted words covered by the mask, and x represents the input text.

[0038] Next, take the example of classifying the emotions of movie reviews mentioned above. In this case, the input text is a movie review text, denoted by [x]. Create the following template:

[0039] It is a [MASK] movie,[x]. In the above template, the mask text is represented by [MASK].

[0040] If the movie review text is "The movie is really boring," the mask text is inserted into the movie review text, and the following predicted text is generated.

[0041] It is a [MASK] movie, The movie is really boring. The text to be predicted is input to the pre-trained language model, the word at the mask text position is predicted by the pre-trained language model, and the predicted word at the [MASK] position is called "boring." Since "boring" is a label word in the Negative label, this movie review text is a negative movie review text.

[0042] The embodiment of the present invention is not limited to the type of template. Also, the embodiment of the present invention may construct various different templates. A template and a language expresser are called a template-language expresser pair (pattern-verbalizer). As described below, a pre-training language model can be trained for each PVP.

[0043] There is no strict order of execution between steps 11 and 12. Step 11 may be executed first and then step 12, or step 12 may be executed first and then step 11. Furthermore, steps 11 and 12 may be executed simultaneously.

[0044] In step 13, a pre-training language model is trained using a first training set including a plurality of labeled first training samples, the training including: generating a first predicted text based on the template using the first training samples as input text; generating a word vector of a predicted word in the first predicted text using the pre-training language model; determining a predicted label of the first training sample based on a similarity between the word vector of the predicted word and a word average vector for each label; and optimizing the pre-training language model based on the predicted label of the first training sample and the labeled label.

[0045] Here, the pre-trained language model is usually obtained by training in advance with a huge data set, including, but not limited to, any one of BERT model, Albert model, Roberta model, XLnet model, etc. In an embodiment of the present invention, when training the pre-trained language model, the first training sample is used as an input text, and a masking text is inserted into the first training sample based on the template to generate a first predicted text. The first predicted text is input into the pre-trained language model, and a word vector of a predicted word in the first predicted text from which the pre-trained language model is generated is obtained.

[0046] Then, calculate the similarity between the word vector of the predicted word and the word average vector of each label, and determine the predicted label of the first training sample according to the similarity. For example, calculate the distance between the word vector of the predicted word and the word average vector of each label, and select the minimum distance; the label corresponding to the minimum distance is the predicted label of the first training sample. Of course, if the calculated predicted word is exactly the label word in the label word space, the predicted word can be directly determined to be the label word.

[0047] Preferably, the embodiment of the present invention sets a distance threshold, and if the minimum distance is equal to or greater than the distance threshold, the label corresponding to the minimum distance is taken as the predicted label of the first training sample; if the minimum distance is greater than the distance threshold, it is determined that there is no label corresponding to the first training sample, so that a predicted word with too low similarity is avoided from being determined as a certain label.

[0048] Then, optimize the pre-training language model based on the predicted label and the labeled label of the first training sample. For example, calculate a loss function based on the predicted label and the labeled label of the first training sample. Optimize model parameters by back propagation. Specific loss function settings, parameter optimization, and training termination conditions can be realized by referring to related prior art, and the description is omitted here.

[0049] Through the above steps, when training a model, the embodiment of the present invention determines the label of a predicted word based on the similarity between the predicted word and the word average vector of the label word for each label, without determining the label of the predicted word based on whether the predicted predicted word exists in the label space. In this way, even if the predicted word does not exist in the label space, it can be accurately classified based on the similarity with the word average vector for each label. This allows more samples to be mapped to labels, reduces the impact of insufficient coverage of the label words, and improves the performance of the pre-trained language model.

[0050] For example, a simple language expression {science} → science means that in the prior art, in the estimation process, a predicted word labeled as [MASK] is judged to be correct only when it is predicted as "science", and the prediction result is judged to be incorrect even though other science-related words such as "physics" and "arithmetic" belong to the science category. The method according to the embodiment of the present invention calculates the similarity between the words "physics" and "science" and finds that the similarity between the two words is high. As a result, the predicted word labeled with [MASK] can be accurately classified even if it is not in the label word space.

[0051] As another example, in the scenario applied to the classification of movie reviews described above, assume that the word located in [MASK] is predicted to be Wonderful, but Wonderful does not exist in the labels Positive or Negative. In contrast, in the case of an embodiment of the present invention, by calculating the similarity between "Wonderful" and the word average vector of the "Positive" label, it is found that the similarity between the two is high. This allows the word to be correctly classified as Positive even if Wonderful does not exist in the Positive label word space.

[0052] An embodiment of the present invention uses a first training corpus to train a pre-training language model, and then uses the pre-training language model to generate a second training corpus that includes more training samples.

[0053] Specifically, an embodiment of the present invention obtains a plurality of unlabeled second training samples; determines predicted labels for the second training samples using the pre-training language model; and labels the second training samples based on the determined predicted labels for the second training samples to generate a second training collection.

[0054] That is, embodiments of the present invention train a pre-training language model on a small sample training set (first training set), and then use the model to label a large unlabeled training set of samples to generate labels (soft labels) for each training sample to obtain a larger second training set. Furthermore, embodiments of the present invention use the second training set to train a final pre-training language model, thereby expanding the training set and training the model to better solve the small sample problem.

[0055] The embodiment of the present invention constructs a plurality of said templates. Different predicted texts are generated according to the templates. Here, different masking texts are generated according to the templates, and / or the positions of the masking texts inserted into the input text according to the templates are different. Then, for each of the templates, one pre-trained language model is trained using the first training set to obtain a plurality of pre-trained language models. In this way, the template and the language expressor form a template-language expressor pair (PVP). By training one pre-trained language model for each PVP, the pre-trained language model is combined with the pre-trained language models trained based on different templates to generate a second training set, thereby making full use of the knowledge learned using each pre-trained language model and improving the accuracy of labeling. Since the reliability of one PVP may be insufficient, the embodiment of the present invention combines the results of multiple PVPs to obtain a predicted label with a relatively high reliability during labeling.

[0056] There are different ways to determine the predicted labels of the second training samples using the trained pre-training language models.

[0057] For example, using each pre-training language model, a probability for each label of a second training sample is determined; and the label with the highest probability is taken as the predicted label for the second training sample.

[0058] Also, for example, a test set is used to measure an accuracy index of label prediction for each pre-training language model, and a weight is set for the accuracy index for each pre-training language model based on the accuracy index. The weight has a positive correlation with the accuracy index. That is, if the accuracy index is high, the weight is large, and conversely, if the accuracy index is low, the weight is small. Using each pre-training language model, a probability for each label of the second training sample is determined. Then, for each label, the probability for the label of the second training sample determined for each pre-training language model is weighted and added based on the weight of the pre-training language model, and a probability statistic value for the label of the second training sample is calculated. Then, the label with the largest probability statistic value is set as the predicted label of the second training sample.

[0059] Furthermore, after generating a second training set based on a plurality of pre-trained language models, a final pre-trained language model is trained using the second training set. In addition, in the embodiment of the present invention, when training the final pre-trained language model, a model having the best label prediction accuracy index is selected from the plurality of pre-trained language models trained using the first training set, and the model is trained using the second training set based on this model to obtain a final pre-trained language model. In this way, the performance of the model obtained by the final training can be improved.

[0060] In an embodiment of the present invention, the pre-trained language model is obtained by training on a huge data set. For example, a pre-trained language model such as BERT (Bidirectional Encoding Representation of Transformers) is a large-scale neural network architecture with a large number of parameters ranging from 100 million to 300 million or more. If the pre-trained language model is trained from scratch on a small data set, overfitting may occur. Therefore, starting from a pre-trained language model trained on a huge data set, the model is usually trained using the second training set by the above method. This training process is also called model fine-tuning. For example, a model is trained using a conventional fine tuning method to obtain a standard classification model.

[0061] An embodiment of the present invention includes multiple fine-tuning schemes when training the model using the second training corpus.

[0062] For example, embodiments of the present invention can train the entire architecture of the pre-trained language model, i.e., train the entire pre-trained language model using the second training set and provide the output of the pre-trained language model to the classifier (softmax layer), where errors are propagated backwards through the entire architecture and the pre-trained weighting parameters of the pre-trained language model are updated during training based on the second training set.

[0063] Also, for example, the embodiment of the present invention maintains the neural network layer of the first part of the pre-trained language model while training the neural network layer of the second part. The neural network layer of the second part is the remaining neural network layer in the pre-trained language model excluding the neural network layer of the first part. That is, another method of using the pre-trained language model is a partial training method. Specifically, while maintaining the weights of the initial neural network layer of the pre-trained language model, only the upper neural network layers are repeatedly trained. The embodiment of the present invention obtains a model with better performance by measuring different maintenance layers and training layers.

[0064] Further, for example, an embodiment of the present invention may keep the entire architecture of the pre-trained language model, i.e., keep all layers of the pre-trained language model and add some new neural network layers to form a new model, and then train the new model using a second training set. During the training process, only the weights of the newly added neural network layers are updated.

[0065] According to the above-mentioned method, an embodiment of the present invention further provides an apparatus for implementing the method. As shown in Figure 2, the model training apparatus according to the embodiment of the present invention includes the following modules:

[0066] The calculation module 21 obtains a plurality of label words for different labels, and calculates an average vector of the label words for each label to obtain a word average vector.

[0067] The construction module 22 constructs a template for generating predictive text by inserting masking text, including predicted words covered by masking, into the input text.

[0068] The first training module 23 trains a pre-training language model using a first training ensemble including a plurality of labeled first training samples.

[0069] The training includes generating a first predicted text based on the template using the first training sample as input text; generating word vectors of predicted words in the first predicted text using the pre-training language model; determining predicted labels of the first training sample based on a similarity between the word vectors of the predicted words and word average vectors for each label; and optimizing the pre-training language model based on the predicted labels of the first training sample and the labeled labels.

[0070] The embodiment of the present invention is able to map more samples to labels by using the above-mentioned modules, without being limited to label words in the label word space.

[0071] Preferably, when determining the predicted label of the first training sample based on the similarity between the word vector of the predicted word and the word average vector for each label, the first training module calculates the distance between the word vector of the predicted word and the word average vector for each label, selects the minimum distance, and sets the label corresponding to the minimum distance as the predicted label of the first training sample.

[0072] Also, preferably, the apparatus further includes: a first acquisition module for obtaining a plurality of unlabeled second training samples; and a sample generation module for determining predicted labels of the second training samples using the pre-training language model; and labeling the second training samples based on the determined predicted labels of the second training samples to generate a second training collection.

[0073] Also, preferably, the construction module further constructs a plurality of templates for generating different predictive texts depending on the template, the first training module trains a pre-training language model for each of the templates using the first training set to obtain a plurality of pre-training language models, and the sample generation module includes a first label generation sub-module that determines a probability for each label of a second training sample for each pre-training language model; and sets the label with the maximum probability as the predicted label of the second training sample.

[0074] Preferably, the construction module further constructs a plurality of templates for generating different predictive texts according to the templates, and the first training module further trains one pre-trained language model using the first training set for each of the templates to obtain a plurality of pre-trained language models, and the sample generation module includes a second label generation sub-module that measures an accuracy index of label prediction for each pre-trained language model using a test set, sets a weight for each pre-trained language model that is positively correlated with the accuracy index based on the accuracy index, and determines a probability for each label of a second training sample using each pre-trained language model; weights and adds the probabilities for the label of the second training sample determined for each pre-trained language model based on the weight of the pre-trained language model for each label, to obtain a probability statistic for the label of the second training sample; and determines the label with the maximum probability statistic as the predicted label of the second training sample.

[0075] Preferably, the apparatus includes a second training module for training a final pre-training language model using the second training corpus.

[0076] 3 is a diagram showing a hardware configuration block of a model training apparatus according to an embodiment of the present invention. As shown in FIG. 3, the model training apparatus 300 includes a processor 302 and a memory 304 in which a computer program is stored.

[0077] The method includes causing the processor to execute the computer program to obtain a plurality of labeled words at different labels and calculate an average vector of the labeled words for each label to obtain a word average vector; constructing a template for generating a predicted text by inserting a masked text including the predicted words covered by masking into an input text; and training a pre-training language model using a first training set including a plurality of labeled first training samples.

[0078] The training includes the steps of generating a first predicted text based on the template using the first training sample as input text; generating a word vector of a predicted word in the first predicted text using the pre-training language model; determining a predicted label of the first training sample based on the similarity between the word vector of the predicted word and a word average vector for each label; and optimizing the pre-training language model based on the predicted label of the first training sample and the labeled labels.

[0079] Furthermore, as shown in FIG. 3, the model training device 300 also includes a network interface 301 , an input device 303 , a hard disk 305 and a display device 306 .

[0080] These interfaces and devices are connected to each other by a bus architecture. The bus architecture includes any number of buses and bridges connected to each other. Specifically, the bus architecture connects various circuits of one or more central processing units (CPUs) represented by the processor 302 and / or graphics processors (GPUs) to various circuits of one or more storage devices represented by the memory 304. The bus architecture also connects various other circuits such as peripheral devices, voltage regulators, and power management circuits. The bus architecture enables communication by connecting these components. In addition to a data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus. These are well-known technologies, so detailed explanations will be omitted here.

[0081] The network interface 301 is connected to a network (eg, the Internet, a local area network, etc.), receives data such as training data from the network, and stores the received data in the hard disk 305 .

[0082] The input device 303 receives various commands input by an operator and transmits them to the processor 302 for execution. The input device 303 includes a keyboard, a pointing device (e.g., a mouse, a trackball, a touch panel or a touch screen, etc.).

[0083] The display device 306 displays the results of the processor 302 executing instructions, such as training progress.

[0084] The memory 304 stores programs and data necessary for the operation of the operating system, as well as intermediate results of calculations performed by the processor 302 and other data.

[0085] In an embodiment of the invention, memory 304 may be volatile or nonvolatile memory, or may include both volatile and nonvolatile memory. Among these, nonvolatile memory may be ROM, PROM, EPROM, EEPROM, or flash memory. Volatile memory may be RAM, which acts as an external cache. Memory 304 for use with the apparatus and methods described herein may include, but is not limited to, these and any other suitable types of memory.

[0086] In some implementations, memory 304 stores an operating system (OS) 3041 and application programs (API) 3042, which may be executable modules or data structures, or a sub-collection or extension thereof.

[0087] Among them, the operating system 3041 includes various system programs such as a framework layer, a kernel library layer, and a driver layer for implementing various core business processes and hardware-based task processing, and the application program 3042 includes various applications such as a browser for implementing various application processes. The application program 3042 includes a program for implementing the method according to the embodiment of the present invention.

[0088] The method disclosed in the embodiment of the present invention is applied to or implemented by the processor 302. The processor 302 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the method is completed by the processor 302's hardware integrated logic circuit or instructions in the form of software. The processor 302 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-fail programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and may implement or execute each method, step, and logic block diagram disclosed in the embodiment of the present invention. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention may be directly executed by a hardware decode processor, or may be executed by a combination of the hardware and software modules of the decode processor. The software modules are stored in a storage medium known in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is in the memory 304. The processor 302 reads information from the memory 304 and implements the steps of the method in conjunction with hardware.

[0089] The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. In a hardware implementation, the processor unit may be implemented as one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units used to perform the functions described in the present invention, or a combination thereof.

[0090] In a software implementation, the techniques are implemented with modules (e.g., procedures, functions, and so on) that perform the functions described above. The software code is stored in a memory and executed by a processor. The memory may be implemented within or external to the processor.

[0091] Specifically, the computer program is executed by the processor 302 to realize the steps of calculating the distance between the word vector of the predicted word and the word average vector for each label, selecting the minimum distance, and setting the label corresponding to the minimum distance as the predicted label of the first training sample.

[0092] In particular, when the computer program is executed by the processor 302, the following steps are realized: after training a pre-training language model using a first training set, a second training sample of a plurality of unlabeled labels is obtained; using the pre-training language model, a predicted label of the second training sample is determined; and based on the determined predicted label of the second training sample, the second training sample is labeled to generate a second training set.

[0093] In particular, when the computer program is executed by the processor 302, the steps of constructing a plurality of templates, each of which generates different predictive text, are realized; training one pre-training language model for each of the templates using the first training set to obtain a plurality of pre-training language models; and determining a predicted label of the second training sample using the pre-training language models includes determining a probability for each label of the second training sample using each pre-training language model; and setting the label with the maximum probability as the predicted label of the second training sample.

[0094] In particular, when the computer program is executed by the processor 302, the steps of constructing a plurality of templates for generating different predictive texts according to the templates are realized; for each of the templates, a pre-trained language model is trained using the first training set to obtain a plurality of pre-trained language models; determining a predicted label of the second training sample using the pre-trained language models includes measuring an accuracy index of label prediction for each pre-trained language model using a test set, setting a weight that is positively correlated with the accuracy index for each pre-trained language model based on the accuracy index, and determining a probability for each label of the second training sample using each pre-trained language model; for each label, weighting and adding the probabilities for the label of the second training sample determined for each pre-trained language model based on the weight of the pre-trained language model, and calculating a probability statistic for the label of the second training sample; and determining the label with the largest probability statistic as the predicted label of the second training sample.

[0095] In particular, the computer program is executed by the processor 302 to realize the steps of generating the second training set and then training a final pre-training language model using the second training set.

[0096] The elements and algorithm steps of the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. This can be easily conceived by those skilled in the art. Whether these functions are implemented by hardware or software depends on the application or design constraints of the invention. Those skilled in the art can appropriately use different methods to implement the functions according to the specific application of the invention, but the implementation should not exceed the scope of the invention.

[0097] It is obvious to those skilled in the art that the specific operations of the above systems, devices and units correspond to the steps of the methods in the above embodiments, and for convenience and brevity, the description is omitted here.

[0098] The devices and methods disclosed in the embodiments of the present invention may be realized in other ways. For example, the devices according to the above-mentioned embodiments are merely examples, and the division of the units is a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or elements may be combined or integrated into another system. Alternatively, some features may be ignored or not implemented. Note that the coupling or direct coupling or communicative connection shown or described is performed via an interface. The indirect coupling or communicative connection of the devices or units may be electrical, mechanical, or another form.

[0099] The units described as separate components may or may not be physically separated. The display components as units may or may not be physical units, i.e., they may be located in one place or distributed among multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the embodiments of the present invention.

[0100] In addition, each functional unit according to each embodiment of the present invention may be integrated into one processing unit, each unit may be a single physical entity, or two or more units may be integrated into one unit.

[0101] The above-mentioned functions may be realized in the form of software functional units and stored in a computer-readable storage medium when sold or used as an independent product. In this case, the part that contributes to the present invention or the prior art or the part of the invention is essentially expressed in the form of a software product. This computer software product, when stored in a storage medium, causes a computer device (such as a personal computer, a server, or a network device) to execute all or part of the steps of the method according to each embodiment of the present invention. The storage medium is a medium capable of storing various program codes, such as a USB memory, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0102] The above describes the specific implementation of the present invention, but the scope of protection of the present invention is not limited thereto. Modifications or replacements within the scope of the disclosed invention can be easily conceived by those skilled in the art, and all of them belong to the scope of the present invention. Therefore, the scope of the present invention is governed by the claims.

Claims

1. A step of acquiring a plurality of label words for different labels, and calculating an average vector of the label words for each label to acquire a word average vector; constructing a template, the template being used to insert masking text into the input text to generate predictive text, the masking text including predicted words that will be covered by the masking; training a pre-training language model using a first training corpus including a plurality of labeled first training samples; The step of training the pre-training language model comprises: generating a first predicted text based on the template using the first training sample as input text; generating word vectors of predicted words in the first predicted text using the pre-training language model, and determining predicted labels for the first training sample based on a similarity between the word vectors of the predicted words and a word mean vector of each label; optimizing the pre-training language model based on the predicted labels and the labeled labels of the first training samples.

2. Determining a predicted label for the first training sample based on a similarity between a word vector of the predicted word and a word mean vector of each label, Calculating the distance between the word vector of the predicted word and the word mean vector of each label, and selecting the minimum distance; 2. The method of claim 1, further comprising: taking the label corresponding to the smallest distance as the predicted label for the first training sample.

3. After training one pre-training language model using the first training corpus, obtaining a second training sample of unlabeled labels; determining predicted labels for the second training samples using the pre-training language model; 2. The method of claim 1, further comprising: labeling the second training samples based on the determined predicted labels of the second training samples to generate a second training corpus.

4. Constructing a plurality of said templates; Generate different predictive texts according to different templates, training a pre-training language model using the first training corpus to obtain a plurality of pre-training language models for each of the templates; Determining predicted labels for the second training samples using the pre-training language model comprises: determining a probability for each label of a second training sample with each pre-training language model; and determining the label with the highest probability as the predicted label for the second training sample.

5. Constructing a plurality of said templates; Generate different predictive texts according to different templates, training a pre-training language model using the first training corpus to obtain a plurality of pre-training language models for each of the templates; Determining predicted labels for the second training samples using the pre-training language model comprises: measuring an accuracy index of label prediction for each pre-training language model using a test corpus, and setting a weight for each pre-training language model that is positively correlated with the accuracy index based on the accuracy index; determining a probability for each label of a second training sample using each pre-training language model; For each label, weighting and adding the probabilities of the second training sample in the label determined for each pre-training language model based on the weights of the pre-training language model, and calculating a probability statistic for the label of the second training sample; and determining the label with the largest probability statistic as the predicted label for the second training sample.

6. 6. The method of claim 3, further comprising, after generating the second training corpus, training a final pre-training language model using the second training corpus.

7. A calculation module that obtains a plurality of label words in different labels, calculates an average vector of the label words for each label, and obtains a word average vector; a construction module for constructing a template, the template being used to generate a predictive text by inserting a masking text into an input text, the masking text including predicted words that are to be covered by the masking; a first training module for training a pre-training language model using a first training corpus including a plurality of labeled first training samples; The first training module includes: generating a first predicted text based on the template using the first training sample as input text; generating word vectors of predicted words in the first predicted text using the pre-training language model, and determining predicted labels of the first training sample based on a similarity between the word vectors of the predicted words and a word mean vector of each label; A model training device comprising: an apparatus for optimizing the pre-training language model based on the predicted labels and the labeled labels of the first training samples.

8. The first training module, when determining the predicted label of the first training sample based on a similarity between the word vector of the predicted word and the word mean vector of each label, Calculate the distance between the word vector of the predicted word and the word mean vector of each label, and select the minimum distance; The apparatus for training a model according to claim 7, characterized in that the label corresponding to the minimum distance is set as the predicted label of the first training sample.

9. a first acquisition module for acquiring a second training sample of a plurality of unlabeled labels; and a sample generation module that uses the pre-training language model to determine predicted labels for the second training samples, and label the second training samples based on the determined predicted labels for the second training samples to generate a second training corpus.

10. the construction module constructs a plurality of the templates; Generate different predictive texts according to different templates, the first training module trains a pre-training language model using the first training corpus to obtain a plurality of pre-training language models for each of the templates; The sample generation module includes: and a first label generation submodule for determining a probability for each label of a second training sample by each pre-training language model, and setting the label with the highest probability as a predicted label for the second training sample.

11. the construction module constructs a plurality of the templates; Generate different predictive texts according to different templates, the first training module trains a pre-training language model using the first training corpus to obtain a plurality of pre-training language models for each of the templates; The sample generation module includes: and a second label generation submodule that uses a test set to measure an accuracy index of label prediction for each preparatory training language model, sets a weight for each preparatory training language model based on the accuracy index so that the weight is positively correlated with the accuracy index, determines a probability for each label of a second training sample using each preparatory training language model, performs a weighted sum of the probabilities for the label of the second training sample determined for each preparatory training language model based on the weight of the preparatory training language model for each label, calculates a probability statistic for the label of the second training sample, and sets a label with the largest probability statistic as a predicted label of the second training sample.

12. 12. The apparatus of claim 9, further comprising a second training module for training a final pre-training language model using the second training corpus.

13. A computer-readable storage medium having a computer program stored therein, the computer program being executed by a processor to realize steps of the model training method according to any one of claims 1 to 5.

Images