Model training method and apparatus, electronic device, and storage medium

Abstract

The application discloses a model training method and device, electronic equipment and a storage medium; the application can obtain a pre-training model and a target model; obtain an unlabeled sample group and a labeled sample group, and unlabeled probability and labeled probability; based on the unlabeled probability and the labeled probability, linearly combine the unlabeled sample group and the labeled sample group respectively to obtain unlabeled augmented samples corresponding to the unlabeled sample group and labeled augmented samples corresponding to the labeled sample group; based on the unlabeled augmented samples and the pre-training model, preliminarily train the target model; based on the labeled augmented samples and the pre-training model, retrain the preliminarily trained target model to obtain a trained target model. In the application, the target model is trained by augmenting the unlabeled samples and the labeled samples and combining the pre-training model, so that the generalization and robustness of the target model can be improved. Therefore, the application can improve the performance of the target model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a model training method, apparatus, electronic device, and storage medium. Background Technology

[0002] To extract useful information from increasingly large and redundant datasets, the required scale of deep learning models is also increasing, and it may even be necessary to integrate multiple deep learning models. However, the slow inference speed and high resource requirements of large-scale pre-trained models (e.g., larger memory capacity, higher performance GPU memory) make them unsuitable for deployment in specific tasks.

[0003] Therefore, knowledge distillation, a model compression method, has been proposed. Knowledge distillation can transfer the knowledge learned by a large-scale pre-trained model to the target model, enabling the target model to perform the tasks that the large-scale pre-trained model could originally achieve.

[0004] However, the performance of target models after knowledge transfer is currently poor. Summary of the Invention

[0005] This application provides a model training method, apparatus, electronic device, and storage medium that can improve the performance of a target model.

[0006] This application provides a model training method, including:

[0007] Obtain the pre-trained model and the target model;

[0008] Obtain unlabeled sample groups and labeled sample groups, as well as unlabeled probability and labeled probability. The unlabeled sample group includes at least two unlabeled samples, and the labeled sample group includes at least two labeled samples.

[0009] Based on the unlabeled probability and the labeled probability, the unlabeled sample group and the labeled sample group are linearly combined to obtain the unlabeled amplified sample corresponding to the unlabeled sample group and the labeled amplified sample corresponding to the labeled sample group.

[0010] Based on unlabeled amplified samples and a pre-trained model, the target model is initially trained to obtain the pre-trained target model.

[0011] Based on labeled augmented samples and pre-trained models, the target model after initial training is retrained to obtain a well-trained target model.

[0012] This application also provides a model training method apparatus, including:

[0013] The first acquisition unit is used to acquire the pre-trained model and the target model;

[0014] The second acquisition unit is used to acquire unlabeled sample groups and labeled sample groups, as well as unlabeled probability and labeled probability. The unlabeled sample group includes at least two unlabeled samples, and the labeled sample group includes at least two labeled samples.

[0015] The amplification unit is used to linearly combine the unlabeled sample group and the labeled sample group based on the unlabeled probability and the labeled probability, respectively, to obtain the unlabeled amplified sample corresponding to the unlabeled sample group and the labeled amplified sample corresponding to the labeled sample group.

[0016] The preliminary training unit is used to perform preliminary training on the target model based on unlabeled augmented samples and a pre-trained model, so as to obtain the target model after preliminary training.

[0017] The retraining unit is used to retrain the target model after initial training based on labeled augmented samples and pre-trained models to obtain a trained target model.

[0018] In some embodiments, the pre-trained model includes multiple network modules, the target model includes multiple network layers, and the network modules correspond one-to-one with the network layers; the labeled augmented samples include labels; the retraining unit is specifically used for:

[0019] The labeled and amplified samples are input into the pre-trained model to obtain the first output result of the target network module, which can be any one of multiple network modules.

[0020] The labeled and amplified samples are input into the target model after initial training to obtain the second output result of the target network layer. The target network layer is the network layer that corresponds to the target network module among multiple network layers.

[0021] Based on the first output, the second output, and the labels, the parameters of the target model are updated again.

[0022] In some embodiments, the retraining unit is specifically used for:

[0023] Based on the first and second output results, the loss of the target network layer is calculated to obtain the network layer loss value.

[0024] Based on the network layer loss values, the parameters from the input layer to the target network layer in the target model are updated;

[0025] Based on the second output and the label, the loss of the target model is calculated to obtain the output loss value;

[0026] Based on the output loss value, the parameters of the target model are updated again.

[0027] In some embodiments, the label includes a first label and a second label, and the retraining unit is specifically used for:

[0028] Based on the first label and the second output result, the loss of the target model is calculated to obtain the first output loss value;

[0029] Based on the second label and the second output result, the loss of the target model is calculated to obtain the second output loss value;

[0030] The first output loss value and the second output loss value are fused to obtain the output loss value.

[0031] In some embodiments, the annotation probability includes a first annotation probability and a second annotation probability, and the retraining unit is specifically used for:

[0032] The first output loss value is weighted using the first labeled probability to obtain the first weighted loss value.

[0033] The second output loss value is weighted using the second labeled probability to obtain the second weighted loss value.

[0034] The output loss value is obtained based on the first weighted loss value and the second weighted loss value.

[0035] In some embodiments, the network layer loss value includes a feature extraction loss value, the first output result includes a first feature, and the second output result includes a second feature; the retraining unit is specifically used for:

[0036] When the dimensions of the first feature and the second feature are inconsistent, a spatial transformation is performed on the first feature to obtain the transformed first feature;

[0037] Calculate the eigennorm 2 between the eigenvalues ​​of the second feature and the eigenvalues ​​of the transformed first feature;

[0038] The feature extraction loss value is determined based on the feature L2 norm.

[0039] In some embodiments, the network layer loss value includes an attention loss value, the first output result includes a first attention probability distribution, and the second output result includes a second attention probability distribution; the retraining unit is specifically used for:

[0040] Calculate the relative attention entropy between the first attention probability distribution and the second attention probability distribution;

[0041] The attention loss value is determined based on the relative entropy of attention.

[0042] In some embodiments, the network layer loss value includes a classification loss value, the first output result includes a first classification probability distribution, and the second output result includes a second classification probability distribution; the retraining unit is specifically used for:

[0043] Calculate the relative entropy between the first and second classification probability distributions;

[0044] The classification loss value is determined based on the relative entropy of classification.

[0045] In some embodiments, the pre-trained model includes multiple network modules, and the target model includes multiple network layers, with a one-to-one correspondence between network modules and network layers; the preliminary training unit is specifically used for:

[0046] The unlabeled amplified samples are input into the pre-trained model to obtain the third output result of the target network module, which can be any one of multiple network modules.

[0047] Input the unlabeled amplified sample into the target model to obtain the fourth output result of the target network layer. The target network layer is the network layer that corresponds to the target network module among multiple network layers.

[0048] Based on the third and fourth output results, the parameters of the target model are initially updated.

[0049] In some embodiments, the first acquisition unit is specifically used for:

[0050] Obtain the pre-trained model and the target model to be processed, which includes multiple network layers to be processed;

[0051] The pre-trained model is divided into blocks to obtain multiple network modules, and each network module corresponds one-to-one with the network layer to be processed.

[0052] Obtain the parameters of the nth network layer in the network module, where n is a positive integer;

[0053] The parameters of the nth network layer are used as the initial parameters of the corresponding network layer to be processed to obtain the target model.

[0054] In some embodiments, the first acquisition unit is specifically used for:

[0055] When the dimensions of the nth network layer are inconsistent with those of the network layer to be processed, the parameters of the network layer to be processed are randomly initialized to obtain the randomly initialized network layer.

[0056] The randomly initialized network layers are trained with masks to obtain masked network layers. The target model includes the masked network layers.

[0057] In some embodiments, the amplification unit is specifically used for:

[0058] Based on the unlabeled probability, the unlabeled sample groups are linearly combined to obtain the unlabeled amplified samples corresponding to the unlabeled sample groups.

[0059] Based on the annotation probability, the annotated sample groups are linearly combined to obtain the annotated augmented samples corresponding to the annotated sample groups;

[0060] The annotation probabilities include a first annotation probability and a second annotation probability. The annotation sample set includes a first annotation sample and a second annotation sample. Based on the annotation probabilities, the annotation sample set is linearly combined to obtain the corresponding annotation augmentation samples, including:

[0061] The first labeled samples are weighted using the first labeled probability to obtain the first sample matrix;

[0062] The second labeled samples are weighted using the second labeled probability to obtain the second sample matrix;

[0063] The first sample matrix and the second sample matrix are fused to obtain the labeled and amplified samples.

[0064] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the model training methods provided in this application.

[0065] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the model training methods provided in this application.

[0066] This application can obtain a pre-trained model and a target model; obtain unlabeled sample groups and labeled sample groups, as well as unlabeled probabilities and labeled probabilities; based on the unlabeled probabilities and labeled probabilities, linearly combine the unlabeled sample groups and labeled sample groups respectively to obtain unlabeled augmented samples corresponding to the unlabeled sample groups and labeled augmented samples corresponding to the labeled sample groups; perform preliminary training on the target model based on the unlabeled augmented samples and the pre-trained model; and perform retraining on the pre-trained target model based on the labeled augmented samples and the pre-trained model to obtain a trained target model.

[0067] In this application, unlabeled and labeled sample groups are linearly combined using unlabeled and labeled probabilities to obtain unlabeled augmented samples and labeled augmented samples. Preliminary training of the target model using the unlabeled augmented samples and a pre-trained model improves the robustness of the target model. Retraining the pre-trained target model using the labeled augmented samples and the pre-trained model improves the generalization ability of the target model. Therefore, this scheme can improve the performance of the target model. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart illustrating the model training method provided in this application;

[0070] Figure 2a This is a schematic diagram of a scenario for the model training method provided in this application;

[0071] Figure 2b This is a schematic diagram of a linear combination of unlabeled samples provided in this application;

[0072] Figure 3 This is a schematic diagram of a model training method device provided in this application;

[0073] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0074] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] This application provides a model training method, apparatus, electronic device, and storage medium.

[0076] Specifically, the model training method device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, desktop computer, smart TV, smart vehicle terminal, etc.; the server can be a single server or a server cluster consisting of multiple servers. In some embodiments, the server can also be implemented in the form of a terminal.

[0077] In some embodiments, the model training method apparatus can also be integrated into multiple electronic devices. For example, the model training method apparatus can be integrated into a terminal and a server, and the terminal and server can jointly implement the model training method of this application.

[0078] In some embodiments, the electronic device can acquire a pre-trained model and a target model; acquire unlabeled sample groups and labeled sample groups, as well as unlabeled probabilities and labeled probabilities; perform linear combinations of the unlabeled sample groups and labeled sample groups based on the unlabeled probabilities and labeled probabilities respectively to obtain unlabeled augmented samples corresponding to the unlabeled sample groups and labeled augmented samples corresponding to the labeled sample groups; perform preliminary training on the target model based on the unlabeled augmented samples and the pre-trained model; and perform retraining on the pre-trained target model based on the labeled augmented samples and the pre-trained model to obtain a trained target model.

[0079] In this application, unlabeled and labeled sample groups are linearly combined using unlabeled and labeled probabilities to obtain unlabeled augmented samples and labeled augmented samples. Preliminary training of the target model using the unlabeled augmented samples and a pre-trained model improves the robustness of the target model. Retraining the pre-trained target model using the labeled augmented samples and the pre-trained model improves the generalization ability of the target model. Therefore, this scheme can improve the performance of the target model.

[0080] Artificial intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquisition of knowledge, and use of that knowledge. This technology can enable machines to have functions similar to human perception, reasoning, and decision-making. AI technology mainly includes several major areas such as computer vision, speech processing, natural language processing (NLP), machine learning, and deep learning.

[0081] In some embodiments, the target model trained by this invention can be applied to tasks in the field of Natural Language Processing (NLP). These NLP tasks may include, but are not limited to, named entity recognition, text classification, stemming, lemmatization, word vectorization, part-of-speech tagging, named entity disambiguation, sentiment analysis, text semantic similarity analysis, language identification, and text summarization. For example, named entity recognition is the task of identifying entities with specific meanings in a sentence and classifying them into categories such as names of people, organizations, dates, places, and times. Text classification categorizes text according to certain rules, and named entity recognition is also a text classification task.

[0082] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0083] In this embodiment, a model training method is provided, such as Figure 1 As shown, the specific process of this model training method can be as follows:

[0084] 110. Obtain the pre-trained model and the target model.

[0085] The pre-trained model (Teacher Model) can be a task-independent model obtained from large-scale training data through supervised learning, unsupervised learning, self-supervised learning, or a combination thereof. For example, the pre-trained model can be the BERT (Bidirectional Encoder Representation from Transformers) series, the XLNet model, or other Transformer-based pre-trained models. The training data can be text, text-image pairs, text-video pairs, etc. In some embodiments, the pre-training process of the pre-trained model can include masking training and next-sentence prediction training. Masking training can involve randomly masking a certain number of words (tokens) in the training text during training and calculating the loss value of the masked tokens using a loss function. Next-sentence prediction training can involve taking sentences A and B as input and predicting whether B is the next sentence of A, allowing the model to learn the relationship between the two sentences.

[0086] The target model (Student Model) can be a network structure designed according to a specific task, and the parameters of this network structure have been initialized. In this embodiment, the specific task can be a classification task of different granularities, such as named entity recognition or text classification.

[0087] In some embodiments, obtaining the pre-trained model and the target model may include, but is not limited to:

[0088] i. Obtain the pre-trained model and the target model to be processed, which includes multiple network layers to be processed.

[0089] The target model to be processed is a target model whose parameters have not yet been initialized, meaning that the parameters of the network layer to be processed need to be initialized. In some embodiments, the network layer to be processed may include a feature extraction sub-layer and an attention sub-layer; or the network layer to be processed may be a classification layer (such as a fully connected network and a softmax network).

[0090] In some embodiments, after obtaining the pre-trained model, the pre-trained model can be fine-tuned based on the specific task to be solved; for example, labeled samples related to text classification can be obtained to fine-tune the pre-trained model.

[0091] ii. Divide the pre-trained model into blocks to obtain multiple network modules, each corresponding to a network layer to be processed.

[0092] In some embodiments, the pre-trained model can be divided into blocks based on the number of network layers in the pre-trained model. These network layers may include, but are not limited to, feature extraction sub-layers and attention sub-layers, and the network modules may also be classification layers, input layers, etc. For example, assuming the pre-trained model has 12 network layers, it can be divided into 4 network modules, each containing 3 network layers. Each network module corresponds one-to-one with the network layer to be processed; therefore, there can be 4 network layers to be processed.

[0093] In some embodiments, the pre-trained model can be divided into blocks based on the knowledge learned by the network layers in the pre-trained model. For example, taking the BERT model as an example, the low-level network is better than the high-level network in encoding phrase-level structural information. The low-level network can also learn surface information features, the intermediate network can learn syntactic structure and syntactic information features, and the high-level network can learn semantic information features. Therefore, the BERT model can be divided into three network modules: the low-level network module, the intermediate network module, and the high-level network module.

[0094] iii. Obtain the parameters of the nth layer in the network module, where n can be any positive integer. For example, if each network module consists of 3 layers, you can obtain the parameters of the 3rd (last) layer, the 2nd layer, or the 1st layer.

[0095] iv. Use the parameters of the nth network layer as the initial parameters of the corresponding network layer to be processed to obtain the target model.

[0096] In some embodiments, it can also be determined whether the dimensions of the nth network layer and the network layer to be processed are consistent, where the dimension can refer to the width of the model; when the dimensions of the nth network layer and the network layer to be processed are inconsistent, the parameters of the network layer to be processed are randomly initialized to obtain a randomly initialized network layer; the randomly initialized network layer is subjected to mask training to obtain a mask-trained network layer, and the target model includes the mask-trained network layer.

[0097] 120. Obtain unlabeled sample groups and labeled sample groups, as well as unlabeled probability and labeled probability. The unlabeled sample group includes at least two unlabeled samples, and the labeled sample group includes at least two labeled samples.

[0098] The unlabeled probability λ1 can conform to a probability distribution, for example, λ1 can conform to a beta distribution. The labeled probability λ2 can also conform to a probability distribution, for example, λ2 can also conform to a beta distribution.

[0099] Unlabeled samples refer to samples without labels; labeled samples refer to samples with labels. For example, the sample can be a vector obtained by encoding data such as text, text-image pairs, and text-video pairs. The specific implementation of the encoding process is not limited and can employ one or more combinations of methods such as one-hot encoding, information retrieval (IR) technology, distributed representation, matrix-based distributed representation, clustering-based distributed representation, neural network-based distributed representation (such as Word2vec), and word embedding encoding.

[0100] In some embodiments, two or more samples may be randomly selected from a batch of unlabeled samples to form an unlabeled sample group; or two or more samples may be randomly selected from a batch of labeled samples to form a labeled sample group.

[0101] In some embodiments, before obtaining the unlabeled sample group and the labeled sample group, an unlabeled set and a labeled set can also be obtained. Training data can be obtained from the unlabeled set in multiple batches for encoding processing to obtain multiple batches of unlabeled samples; training data can be obtained from the labeled set in multiple batches for encoding processing to obtain multiple batches of labeled samples. The process of inputting a batch of unlabeled samples and / or labeled samples into the target model for training can be called an iteration. The number of unlabeled samples or labeled samples in a batch (batch size) is not limited. It should be noted that the training process of the target model is a multi-iterative process. Steps 120-150 of this application are only illustrated using a single iteration as an example.

[0102] 130. Based on the unlabeled probability and the labeled probability, perform linear combination on the unlabeled sample group and the labeled sample group respectively to obtain the unlabeled amplified sample corresponding to the unlabeled sample group and the labeled amplified sample corresponding to the labeled sample group.

[0103] The performance of the target model is limited by the number of labeled samples corresponding to the specific task. When the number of labeled samples is small, the performance of the target model obtained by distillation also decreases. Therefore, a linear combination based on the existing samples can be performed to obtain richer samples in order to improve the performance of the distilled target model.

[0104] (1) Based on the unlabeled probability, the unlabeled sample group is linearly combined to obtain the unlabeled amplified sample corresponding to the unlabeled sample group.

[0105] In some embodiments, the unlabeled probability includes a first unlabeled probability and a second unlabeled probability, and the unlabeled sample group includes a first unlabeled sample and a second unlabeled sample. The first unlabeled sample can be weighted using the first unlabeled probability to obtain a first feature matrix; the second unlabeled sample can be weighted using the second unlabeled probability to obtain a second feature matrix; and the first feature matrix and the second feature matrix can be fused to obtain an unlabeled amplified sample.

[0106] For example, the following formula can be used to calculate unlabeled amplified samples:

[0107]

[0108] in, Let E represent unlabeled amplified samples, λ1 represent the first unlabeled probability, (1-λ1) represent the second unlabeled probability, and E 11 E represents the first unlabeled sample. 12 This indicates the second unlabeled sample.

[0109] In some embodiments, the unlabeled sample group includes n unlabeled samples, which can correspond to n unlabeled probabilities. The unlabeled amplified samples can be calculated using the following formula:

[0110]

[0111] Where, λ 1n Let λ represent the probability of the nth unlabeled element. 11 +…+λ 1n =1), E 1n This represents the nth unlabeled sample.

[0112] (2) Based on the annotation probability, the annotation sample group is linearly combined to obtain the annotation amplification sample corresponding to the annotation sample group.

[0113] In some embodiments, the annotation probability includes a first annotation probability and a second annotation probability, and the annotation sample group includes a first annotation sample and a second annotation sample; the first annotation sample is weighted using the first annotation probability to obtain a first sample matrix; the second annotation sample is weighted using the second annotation probability to obtain a second sample matrix; the first sample matrix and the second sample matrix are fused to obtain an annotation augmentation sample.

[0114] For example, the labeled amplified sample can be calculated using the following formula:

[0115]

[0116] in, Let E represent the labeled amplified sample, λ2 represent the first labeling probability, (1-λ2) represent the second labeling probability, and E 21 E represents the first labeled sample. 22 This indicates the second labeled sample.

[0117] In some embodiments, the labeled sample group includes n labeled samples, which may correspond to n labeled probabilities. The labeled augmented samples can be calculated using the following formula:

[0118]

[0119] Where, λ 2n Represents the probability of the nth label (λ) 21 +…+λ 2n =1), E 2n This represents the nth labeled sample.

[0120] It should be noted that a batch of unlabeled or labeled samples can be randomly combined to obtain multiple unlabeled sample groups and / or labeled sample groups. Each unlabeled sample group and / or labeled sample group can be linearly combined to obtain multiple unlabeled amplified samples and / or multiple labeled samples. This embodiment only uses the linear combination of one unlabeled sample group and one labeled sample group as an example for illustration.

[0121] 140. Based on unlabeled amplified samples and pre-trained models, the target model is initially trained to obtain the initially trained target model.

[0122] By linearly combining unlabeled samples, a richer and more diverse range of samples is introduced. Using unlabeled augmented samples to train the target model can improve the robustness of model training.

[0123] In some embodiments, the pre-trained model includes multiple network modules, and the target model includes multiple network layers, with a one-to-one correspondence between network modules and network layers; preliminary training of the target model based on unlabeled augmented samples and the pre-trained model may include, but is not limited to:

[0124] i. Input the unlabeled amplified samples into the pre-trained model to obtain the third output result of the target network module. The target network module is any one of multiple network modules.

[0125] ii. Input the unlabeled amplified sample into the target model to obtain the fourth output result of the target network layer. The target network layer is the network layer that corresponds to the target network module among multiple network layers.

[0126] iii. Based on the third and fourth output results, the parameters of the target model are initially updated.

[0127] In some embodiments, loss calculation is performed on the target network layer based on the third and fourth output results to obtain the network layer loss value; based on the network layer loss value, the parameters from the input layer to the target network layer in the target model are updated. The specific implementation of calculating the loss value of the target network layer based on the third and fourth output results can refer to step 150, which describes "calculating the loss value of the target network layer based on the first and second output results," and will not be repeated here.

[0128] 150. Based on the labeled augmented samples and the pre-trained model, the target model after the initial training is retrained to obtain the trained target model.

[0129] By linearly combining labeled samples, more labeled samples are introduced, enabling the target model to learn knowledge from the pre-trained model from a wider sample space. This reduces its dependence on the amount of labeled sample data, thereby improving the distillation learning effect, such as enhancing the generalization of the target model and avoiding oscillation problems when predicting data outside the sample data distribution.

[0130] In some embodiments, the pre-trained model includes multiple network modules, the target model includes multiple network layers, and the network modules and network layers correspond one-to-one. The labeled augmented samples include labels. In this embodiment, only the target network model is used as an example for illustration. Based on the labeled augmented samples and the pre-trained model, the target model after the initial training is retrained, which may include, but is not limited to, the following steps:

[0131] i. Input the labeled amplified samples into the pre-trained model to obtain the first output result of the target network module.

[0132] The target network module can be any one of multiple network modules; the target network module may include multiple network layers. In some embodiments, the first output result may be the first feature output by the last network layer of the target network module. In some embodiments, each network layer may include an attention sublayer, and the first output result may be the first attention probability distribution corresponding to the attention sublayer. In some embodiments, the target network module may be a classification layer, and the first output result may be a first classification probability distribution.

[0133] ii. Input the labeled amplified samples into the target model after preliminary training to obtain the second output result of the target network layer.

[0134] In this context, the target network layer is the network layer corresponding to the target network module among multiple network layers. In some embodiments, the second output result may be a second feature output by the target network layer. In some embodiments, the target network layer may include an attention sublayer, and the second output result may be a second attention probability distribution corresponding to that attention sublayer. In some embodiments, the target network layer may be a classification layer, and the second output result may be a second classification probability distribution.

[0135] iii. Based on the first output result, the second output result, and the label, update the parameters of the target model again.

[0136] The label can be a label corresponding to the amplified sample. In some embodiments, the label may include a first label and a second label; the first label is the label corresponding to the first labeled sample, and the second label is the label corresponding to the second labeled sample. In some embodiments, the label can be an amplification label obtained by linearly combining the first label and the second label; for example, the amplification label can be obtained using the following formula:

[0137]

[0138] in, Y1 represents the amplification tag, and Y2 represents the second tag.

[0139] In some embodiments, the parameters of the target model are updated again based on the first output result, the second output result, and the label, which may include, but is not limited to:

[0140] ①Based on the first and second output results, the loss of the target network layer is calculated to obtain the network layer loss value.

[0141] In some embodiments, the network layer loss value includes a feature extraction loss value, the first output result includes a first feature, and the second output result includes a second feature; when the dimensions of the first feature and the second feature are inconsistent, the first feature is spatially transformed to obtain the transformed first feature; the feature 2 norm between the feature value of the second feature and the feature value of the transformed first feature is calculated; and the feature extraction loss value is determined based on the feature 2 norm.

[0142] For example, the feature extraction loss value corresponding to the target network layer can be calculated using the following formula:

[0143]

[0144] Among them, L MSE This represents the feature extraction loss value corresponding to the target network layer, and N represents the number of labeled and amplified samples in a batch of labeled and amplified samples. This represents the feature extraction loss value corresponding to the i-th labeled amplified sample.

[0145]

[0146] in, This represents the first feature corresponding to the i-th labeled amplified sample output by the target network module. a represents the second feature corresponding to the i-th labeled amplified sample output by the target network layer; m denoted by , m represents the target network layer corresponding to the target network module; c represents the number of blocks in the network module (i.e., the number of layers in the target model's network); l is the text length corresponding to the labeled augmented sample; j represents the j-th word corresponding to the labeled augmented sample; k represents the dimension of the first feature; i represents the i-th labeled augmented sample in a batch of labeled augmented samples; and W represents the spatial transformation factor.

[0147] In some embodiments, the network layer loss value includes an attention loss value, the first output result includes a first attention probability distribution, and the second output result includes a second attention probability distribution; the relative attention entropy between the first attention probability distribution and the second attention probability distribution is calculated; and the attention loss value is determined based on the relative attention entropy.

[0148] For example, the attention loss value corresponding to the target network layer can be calculated using the following formula:

[0149]

[0150] Among them, L KLA B represents the attention loss value corresponding to the target network, and B represents the number of attention heads in the attention sublayer. This represents the relative entropy corresponding to the b-th attention head.

[0151]

[0152] in, This represents the probability distribution of the first attention. Let represent the second attention probability distribution. The attention probability distribution can be calculated using the following formula:

[0153]

[0154] Where Q represents the Query vector in the attention head, K represents the Key vector in the attention head, and D represents the dimension of the Query vector or Key vector in the attention head.

[0155] In some embodiments, the network layer loss value includes a classification loss value, the first output result includes a first classification probability distribution, and the second output result includes a second classification probability distribution; the classification relative entropy between the first classification probability distribution and the second classification probability distribution is calculated; and the classification loss value is determined based on the classification relative entropy.

[0156] For example, the classification loss value corresponding to the target network layer can be calculated using the following formula:

[0157]

[0158] Among them, L KLP This represents the classification loss value corresponding to the target network layer. Let represent the relative entropy of the classification corresponding to the i-th labeled amplified sample.

[0159]

[0160] in, Y This indicates the number of categories corresponding to the classification layer. This represents the probability distribution for the first category. This represents the probability distribution of the second category.

[0161] ② Based on the network layer loss value, update the parameters from the input layer to the target network layer in the target model.

[0162] The input layer can be defined as the first layer through which the target model processes the input sample.

[0163] In some embodiments, the parameters from the input layer to the target network layer in the target model are updated based on the feature extraction loss value. Therefore, the target model can learn the feature representation of the pre-trained model.

[0164] In some embodiments, the parameters from the input layer to the target network layer in the target model are updated based on the attention loss value. Therefore, the target model can learn the attention distribution of the pre-trained model.

[0165] In some embodiments, the parameters of the target model, from the input layer to the target network layer (classification layer), are updated based on the classification loss value. This allows for more accurate classification by the target model.

[0166] In some embodiments, the feature extraction loss value, attention loss value, and classification loss value are combined in pairs, multiple combinations, or all combinations to obtain a combined loss, and the parameters from the input layer to the target network layer in the target model are updated based on the combined loss.

[0167] ③ Based on the second output result and the label, calculate the loss of the target model to obtain the output loss value.

[0168] In some embodiments, the labels include a first label and a second label. Based on the second output result and the labels, loss calculation is performed on the target model to obtain an output loss value, which may include, but is not limited to, the following steps:

[0169] a. Based on the first label and the second output result, calculate the loss of the target model to obtain the first output loss value.

[0170] For example, the first output loss value can be calculated using the following formula:

[0171]

[0172] Among them, CE(P) S Y1) represents the first output loss value. i,j This indicates the first tag.

[0173] b. Based on the second label and the second output result, calculate the loss of the target model to obtain the second output loss value.

[0174] For example, the second output loss value can be calculated using the following formula:

[0175]

[0176] Among them, CE(P) S Y2) represents the first output loss value. i,j This indicates the second tag.

[0177] c. Perform loss fusion on the first output loss value and the second output loss value to obtain the output loss value.

[0178] In some embodiments, the annotation probability includes a first annotation probability and a second annotation probability. Loss fusion is performed on the first output loss value and the second output loss value to obtain the output loss value, which may include, but is not limited to, the following steps:

[0179] The first output loss value is weighted using the first labeling probability to obtain the first weighted loss value; the second output loss value is weighted using the second labeling probability to obtain the second weighted loss value; and the output loss value is obtained based on the first weighted loss value and the second weighted loss value.

[0180] L MixHard =λ2·CE(P S ,Y1)+(1-λ2)·CE(P S Y2)

[0181] Among them, L MixHard This indicates the output loss value.

[0182] In some embodiments, the annotation probability includes a first annotation probability and a second annotation probability. The amplification label corresponding to the amplified sample can be calculated, and the loss of the target model can be calculated based on the amplification label and the second output result to obtain the output loss value.

[0183] For example, the output loss value can be calculated using the following formula:

[0184]

[0185] in, The amplification tag corresponding to the i-th labeled amplified sample.

[0186] ④ Based on the output loss value, the parameters of the target model are updated again.

[0187] In some embodiments, the specific implementation method for updating the parameters of the target model or network layer in this application is not limited. For example, methods such as gradient descent, batch gradient descent (BGD), and stochastic gradient descent (SGD) can be used.

[0188] The target model trained in this application can be applied to various natural language processing tasks, such as search, public accounts, text classification, and dialogue open platform scenarios.

[0189] As shown above, in this embodiment, the pre-trained model can be divided into blocks to obtain multiple network modules. The parameters of the network layers in the target network module are used as the parameters of the corresponding network layers in the target model to obtain the initialized target model. Then, the unlabeled probability and labeled probability conforming to the probability distribution can be obtained. Based on the unlabeled probability and labeled probability, the unlabeled sample group and the labeled sample group are linearly combined to obtain unlabeled augmented samples and labeled augmented samples, thereby introducing richer samples. Then, the target model is initially trained based on the unlabeled augmented samples and the pre-trained model, which improves the robustness of the target model. The target model is then retrained based on the labeled augmented samples and the pre-trained model, which allows the target model to transfer and learn the knowledge of the pre-trained model in a larger sample space, thereby improving the generalization of the target model. Therefore, this solution can improve the performance of the target model.

[0190] The model training scheme provided in this application can be applied to various scenarios where knowledge is transferred from pre-trained models. For example, ... Figure 2a The diagram shown is a scenario illustration of a model training method provided in this embodiment. Taking the pre-trained model as the BERT model (Teacher Model) and the target model (Student Model) network structure as a Transformer-based network structure as an example, the model training method of this application will be further described in detail.

[0191] The specific process of a model training method is as follows:

[0192] 210. Obtain the pre-trained model and the target model.

[0193] The pre-trained model consists of N network layers. The target model consists of k network layers.

[0194] The pre-trained model can be divided into c = N / k network modules, and the parameters of the last layer of each network module can be used as the initial parameters of the corresponding network layer in the target model.

[0195] Furthermore, when the width of the network layers in the target model is changed, randomly initializing the parameters of the target model can introduce unlabeled samples for masked training of the target model, thereby improving the performance of the target model.

[0196] 220. Obtain the unlabeled set and the labeled set.

[0197] 230. Obtain unlabeled sample groups and unlabeled probabilities. Each unlabeled sample group includes at least two unlabeled samples. Based on the unlabeled probabilities, perform a linear combination of the unlabeled sample groups to obtain the unlabeled amplified samples corresponding to the unlabeled sample groups.

[0198] A batch of training data can be obtained from an unlabeled dataset and encoded to produce a batch of unlabeled samples. These unlabeled samples can then be paired and linearly combined to obtain augmented samples. For example... Figure 2a As shown, taking only two unlabeled samples as an example, E 11 E represents the first unlabeled sample. 12 This represents the second unlabeled sample, based on the unlabeled probability E. 11 and E 12 Linear combination was performed to obtain unlabeled amplified samples.

[0199] Based on unlabeled probability of E 11 and E 12 The implementation method for linear combination can be referred to Figure 2b As shown, λ1 represents the unlabeled probability, and λ1 follows a beta distribution (λ1~Beta(α,α)).

[0200] 240. Based on unlabeled amplified samples and pre-trained models, the target model is initially trained to obtain the initially trained target model.

[0201] Unlabeled augmented samples are input into the pre-trained model to obtain the third output of the target network module, which can be any one of multiple network modules. Unlabeled augmented samples are then input into the target model to obtain the fourth output of the target network layer, which is the network layer corresponding to the target network module. Based on the third and fourth outputs, the loss of the target network layer is calculated to obtain its loss value. Based on this loss value, the parameters from the input layer to the target network layer in the target model are updated.

[0202] For example, such as Figure 2a As shown, taking layer N as the last layer of the target network module and layer k as the target network layer as an example, the feature extraction loss values ​​of layer k and layer N can be calculated. The first feature output by layer N is Feature map1, and the second feature output by layer k is Feature map2. L is calculated based on Feature map1 and Feature map2. MSE Furthermore, the attention loss values ​​of layer k and layer N can be calculated. The first attention probability distribution of the attention sub-layer of layer N is Attention Matrix1, and the second attention probability distribution of the attention sub-layer of layer k is Attention Matrix2. L is calculated based on Attention Matrix1 and Attention Matrix2.KLA Based on L MSE and L KLA Update the parameters of layers 0 through k.

[0203] Taking Classifier1 (classification layer) as the target network module and Classifier2 as the target network layer in the diagram as an example, we can obtain the first classification probability distribution Prediction Distribution1 output by Classifier1, and the second classification probability distribution Prediction Distribution2 output by Classifier2. Based on Prediction Distribution1 and Prediction Distribution2, we can calculate L. KLP Based on L KLP Update the parameters of the target model (layer0-Classifier2).

[0204] 250. Obtain the labeled sample group and the labeled probability. The labeled sample group includes at least two labeled samples. Based on the labeled probability, perform a linear combination of the labeled sample group to obtain the labeled amplified sample corresponding to the labeled sample group.

[0205] Step 250 can be referred to in steps 230, 120 and 130, and will not be repeated here.

[0206] 260. Based on the labeled amplified samples and the pre-trained model, the target model after the initial training is retrained to obtain the trained target model.

[0207] The pre-trained model includes multiple network modules, the target model includes multiple network layers, and the network modules and network layers correspond one-to-one. The labeled augmented samples include labels, which include a first label and a second label, and the labeling probabilities include a first labeling probability and a second labeling probability.

[0208] 1) Input the labeled amplified samples into the pre-trained model to obtain the first output result of the target network module. The target network module is any one of the multiple network modules.

[0209] 2) Input the labeled amplified samples into the target model after preliminary training to obtain the second output result of the target network layer. The target network layer is the network layer that corresponds to the target network module among multiple network layers.

[0210] 3) Based on the first and second output results, calculate the loss of the target network layer to obtain the network layer loss value.

[0211] 4) Update the parameters from the input layer to the target network layer in the target model based on the network layer loss value.

[0212] It should be noted that steps 1)-4) can be referred to steps 240 and 150, and will not be repeated here.

[0213] 5) Calculate the loss of the target model based on the first label and the second output result to obtain the first output loss value; calculate the loss of the target model based on the second label and the second output result to obtain the second output loss value; weight the first output loss value using the first label probability to obtain the first weighted loss value; weight the second output loss value using the second label probability to obtain the second weighted loss value; and obtain the output loss value based on the first weighted loss value and the second weighted loss value.

[0214] For example, such as Figure 2a As shown, the first output loss value is calculated based on Prediction Distribution2 and the first label, and the second output loss value is calculated based on Prediction Distribution2 and the second label Y1. The first and second output loss values ​​are then weighted and summed based on the labeled probability λ2 to obtain L. MixHard .

[0215] 6) Update the parameters of the target model again based on the output loss value.

[0216] As can be seen from the above, in this embodiment, the pre-trained model can be divided into multiple network modules. The parameters of the network layers in the target network module are used as the parameters of the corresponding network layers in the target model to obtain the initialized target model. Then, unlabeled probabilities conforming to a probability distribution can be obtained. Based on the unlabeled probabilities, the unlabeled sample groups are linearly combined to obtain unlabeled augmented samples. The target model is initially trained based on the unlabeled augmented samples and the pre-trained model. By introducing richer unlabeled samples to train the target model, the robustness of the target model is improved. Furthermore, labeled probabilities conforming to a probability distribution can be obtained. Based on the labeled probabilities, the labeled sample groups are linearly combined to obtain labeled augmented samples. The initially trained target model is retrained based on the labeled augmented samples and the pre-trained model. This allows the target model to transfer and learn the knowledge of the pre-trained model in a larger sample space, improving the generalization of the target model. Therefore, this solution can improve the performance of the target model.

[0217] To better implement the above methods, this application also provides a model training method apparatus, which can be specifically integrated into an electronic device. For example, in this embodiment, the method of this application will be described in detail by taking the model training method apparatus specifically integrated into an electronic device as an example.

[0218] For example, such as Figure 3 As shown, the model training method apparatus may include a first acquisition unit 301, a second acquisition unit 302, an augmentation unit 303, a preliminary training unit 304, and a retraining unit 305, as follows:

[0219] (I) First Acquisition Unit 301

[0220] The first acquisition unit 301 is used to acquire the pre-trained model and the target model;

[0221] (II) Second Acquisition Unit 302

[0222] The second acquisition unit 302 is used to acquire an unlabeled sample group and a labeled sample group, as well as an unlabeled probability and a labeled probability. The unlabeled sample group includes at least two unlabeled samples, and the labeled sample group includes at least two labeled samples.

[0223] (III) Amplification Unit 303

[0224] Amplification unit 303 is used to linearly combine unlabeled sample groups and labeled sample groups based on unlabeled probability and labeled probability, respectively, to obtain unlabeled amplified samples corresponding to unlabeled sample groups and labeled amplified samples corresponding to labeled sample groups.

[0225] (IV) Preliminary Training Unit 304

[0226] The preliminary training unit 304 is used to perform preliminary training on the target model based on unlabeled amplified samples and a pre-trained model to obtain the pre-trained target model.

[0227] (V) Retraining Unit 305

[0228] The retraining unit 305 is used to retrain the target model after initial training based on labeled augmented samples and pre-trained models to obtain a trained target model.

[0229] In some embodiments, the pre-trained model includes multiple network modules, the target model includes multiple network layers, and the network modules correspond one-to-one with the network layers. The labeled augmented samples include labels. The retraining unit 305 is specifically used for:

[0230] The labeled and amplified samples are input into the pre-trained model to obtain the first output result of the target network module, which can be any one of multiple network modules.

[0231] The labeled and amplified samples are input into the target model after initial training to obtain the second output result of the target network layer. The target network layer is the network layer that corresponds to the target network module among multiple network layers.

[0232] Based on the first output, the second output, and the labels, the parameters of the target model are updated again.

[0233] In some embodiments, the retraining unit 305 is specifically used for:

[0234] Based on the first and second output results, the loss of the target network layer is calculated to obtain the network layer loss value.

[0235] Based on the network layer loss values, the parameters from the input layer to the target network layer in the target model are updated;

[0236] Based on the second output and the label, the loss of the target model is calculated to obtain the output loss value;

[0237] Based on the output loss value, the parameters of the target model are updated again.

[0238] In some embodiments, the label includes a first label and a second label, and the retraining unit 305 is specifically used for:

[0239] Based on the first label and the second output result, the loss of the target model is calculated to obtain the first output loss value;

[0240] Based on the second label and the second output result, the loss of the target model is calculated to obtain the second output loss value;

[0241] The first output loss value and the second output loss value are fused to obtain the output loss value.

[0242] In some embodiments, the annotation probability includes a first annotation probability and a second annotation probability, and the retraining unit 305 is specifically used for:

[0243] The first output loss value is weighted using the first labeled probability to obtain the first weighted loss value.

[0244] The second output loss value is weighted using the second labeled probability to obtain the second weighted loss value.

[0245] The output loss value is obtained based on the first weighted loss value and the second weighted loss value.

[0246] In some embodiments, the network layer loss value includes a feature extraction loss value, the first output result includes a first feature, and the second output result includes a second feature; the retraining unit 305 is specifically used for:

[0247] When the dimensions of the first feature and the second feature are inconsistent, a spatial transformation is performed on the first feature to obtain the transformed first feature;

[0248] Calculate the eigennorm 2 between the eigenvalues ​​of the second feature and the eigenvalues ​​of the transformed first feature;

[0249] The feature extraction loss value is determined based on the feature L2 norm.

[0250] In some embodiments, the network layer loss value includes an attention loss value, the first output result includes a first attention probability distribution, and the second output result includes a second attention probability distribution; the retraining unit 305 is specifically used for:

[0251] Calculate the relative attention entropy between the first attention probability distribution and the second attention probability distribution;

[0252] The attention loss value is determined based on the relative entropy of attention.

[0253] In some embodiments, the network layer loss value includes a classification loss value, the first output result includes a first classification probability distribution, and the second output result includes a second classification probability distribution; the retraining unit 305 is specifically used for:

[0254] Calculate the relative entropy between the first and second classification probability distributions;

[0255] The classification loss value is determined based on the relative entropy of classification.

[0256] In some embodiments, the pre-trained model includes multiple network modules, and the target model includes multiple network layers, with a one-to-one correspondence between the network modules and network layers; the preliminary training unit 304 is specifically used for:

[0257] The unlabeled amplified samples are input into the pre-trained model to obtain the third output result of the target network module, which can be any one of multiple network modules.

[0258] Input the unlabeled amplified sample into the target model to obtain the fourth output result of the target network layer. The target network layer is the network layer that corresponds to the target network module among multiple network layers.

[0259] Based on the third and fourth output results, the parameters of the target model are initially updated.

[0260] In some embodiments, the first acquisition unit 301 is specifically used for:

[0261] Obtain the pre-trained model and the target model to be processed, which includes multiple network layers to be processed;

[0262] The pre-trained model is divided into blocks to obtain multiple network modules, and each network module corresponds one-to-one with the network layer to be processed.

[0263] Obtain the parameters of the nth network layer in the network module, where n is a positive integer;

[0264] The parameters of the nth network layer are used as the initial parameters of the corresponding network layer to be processed to obtain the target model.

[0265] In some embodiments, the first acquisition unit 301 is specifically used for:

[0266] When the dimensions of the nth network layer are inconsistent with those of the network layer to be processed, the parameters of the network layer to be processed are randomly initialized to obtain the randomly initialized network layer.

[0267] The randomly initialized network layers are trained with masks to obtain masked network layers. The target model includes the masked network layers.

[0268] In some embodiments, the amplification unit 303 is specifically used for:

[0269] Based on the unlabeled probability, the unlabeled sample groups are linearly combined to obtain the unlabeled amplified samples corresponding to the unlabeled sample groups.

[0270] Based on the annotation probability, the annotated sample groups are linearly combined to obtain the annotated augmented samples corresponding to the annotated sample groups;

[0271] The annotation probabilities include a first annotation probability and a second annotation probability. The annotation sample set includes a first annotation sample and a second annotation sample. Based on the annotation probabilities, the annotation sample set is linearly combined to obtain the corresponding annotation augmentation samples, including:

[0272] The first labeled samples are weighted using the first labeled probability to obtain the first sample matrix;

[0273] The second labeled samples are weighted using the second labeled probability to obtain the second sample matrix;

[0274] The first sample matrix and the second sample matrix are fused to obtain the labeled and amplified samples.

[0275] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0276] As can be seen from the above, the model training method apparatus of this embodiment can linearly combine the unlabeled sample group and the labeled sample group using the unlabeled probability and the labeled probability to obtain unlabeled augmented samples and labeled augmented samples. Preliminary training of the target model based on the unlabeled augmented samples and the pre-trained model can improve the robustness of the target model; further training of the pre-trained target model based on the labeled augmented samples and the pre-trained model can improve the generalization ability of the target model. Therefore, this scheme can improve the performance of the target model.

[0277] This application also provides an electronic device. In this embodiment, the electronic device will be described in detail as an example, such as... Figure 4 As shown, it illustrates the structural diagram of the electronic device involved in this application, specifically:

[0278] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0279] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0280] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0281] The electronic device also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0282] The electronic device may also include an input module 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0283] The electronic device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0284] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0285] Obtain the pre-trained model and the target model;

[0286] Obtain unlabeled sample groups and labeled sample groups, as well as unlabeled probability and labeled probability. The unlabeled sample group includes at least two unlabeled samples, and the labeled sample group includes at least two labeled samples.

[0287] Based on the unlabeled probability and the labeled probability, the unlabeled sample group and the labeled sample group are linearly combined to obtain the unlabeled amplified sample corresponding to the unlabeled sample group and the labeled amplified sample corresponding to the labeled sample group.

[0288] Based on unlabeled amplified samples and a pre-trained model, the target model is initially trained to obtain the pre-trained target model.

[0289] Based on labeled augmented samples and pre-trained models, the target model after initial training is retrained to obtain a well-trained target model.

[0290] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0291] As shown above, in this embodiment, the electronic device can linearly combine the unlabeled sample group and the labeled sample group using the unlabeled probability and the labeled probability to obtain unlabeled augmented samples and labeled augmented samples. Preliminary training of the target model based on the unlabeled augmented samples and the pre-trained model can improve the robustness of the target model. Retraining the pre-trained target model based on the labeled augmented samples and the pre-trained model can improve the generalization ability of the target model. Therefore, this solution can improve the performance of the target model.

[0292] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0293] Therefore, this application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the model training methods provided in this application. For example, the instructions can execute the following steps:

[0294] Obtain the pre-trained model and the target model;

[0295] Obtain unlabeled sample groups and labeled sample groups, as well as unlabeled probability and labeled probability. The unlabeled sample group includes at least two unlabeled samples, and the labeled sample group includes at least two labeled samples.

[0296] Based on the unlabeled probability and the labeled probability, the unlabeled sample group and the labeled sample group are linearly combined to obtain the unlabeled amplified sample corresponding to the unlabeled sample group and the labeled amplified sample corresponding to the labeled sample group.

[0297] Based on unlabeled amplified samples and a pre-trained model, the target model is initially trained to obtain the pre-trained target model.

[0298] Based on labeled augmented samples and pre-trained models, the target model after initial training is retrained to obtain a well-trained target model.

[0299] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0300] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the named entity recognition or text classification aspects provided in the above embodiments.

[0301] Since the instructions stored in the storage medium can execute the steps in any of the model training methods provided in this application, the beneficial effects that any of the model training methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0302] The above provides a detailed description of a model training method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, include: Obtain the pre-trained model and the target model; Obtain unlabeled sample groups and labeled sample groups, as well as unlabeled probability and labeled probability, wherein the unlabeled sample group includes at least two unlabeled samples, and the labeled sample group includes at least two labeled samples; Based on the unlabeled probability and the labeled probability, the unlabeled sample group and the labeled sample group are linearly combined to obtain the unlabeled amplified sample corresponding to the unlabeled sample group and the labeled amplified sample corresponding to the labeled sample group; wherein, the labeled amplified sample includes a label; Based on the unlabeled amplified samples and the pre-trained model, the target model is initially trained to obtain the initially trained target model, including: inputting the unlabeled amplified samples into the pre-trained model to obtain a third output result of the target network module, wherein the target network module is any one of multiple network modules; inputting the unlabeled amplified samples into the target model to obtain a fourth output result of the target network layer, wherein the target network layer is the network layer corresponding to the target network module among multiple network layers; calculating the loss of the target network layer based on the third and fourth output results to obtain the network layer loss value; and updating the parameters from the input layer to the target network layer in the target model based on the network layer loss value to obtain the initially trained target model. Based on the labeled augmented samples and the pre-trained model, the initially trained target model is retrained to obtain a trained target model, including: inputting the labeled augmented samples into the pre-trained model to obtain a first output result of the target network module, wherein the target network module is any one of the plurality of network modules; inputting the labeled augmented samples into the initially trained target model to obtain a second output result of the target network layer, wherein the target network layer is the network layer corresponding to the target network module among the plurality of network layers; calculating the loss of the target network layer according to the first output result and the second output result to obtain a network layer loss value; updating the parameters from the input layer to the target network layer in the target model based on the network layer loss value; calculating the loss of the target model according to the second output result and the label to obtain an output loss value; and updating the parameters of the target model again based on the output loss value to obtain a trained target model.

2. The model training method as described in claim 1, characterized in that, The pre-trained model includes multiple network modules, and the target model includes multiple network layers, with each network module corresponding to one of the network layers.

3. The model training method as described in claim 1, characterized in that, The labels include a first label and a second label. The step of calculating the loss of the target model based on the second output result and the labels to obtain an output loss value includes: Based on the first label and the second output result, the loss of the target model is calculated to obtain the first output loss value; Based on the second label and the second output result, the loss of the target model is calculated to obtain the second output loss value; The first output loss value and the second output loss value are fused to obtain the output loss value.

4. The model training method as described in claim 3, characterized in that, The labeled probabilities include a first labeled probability and a second labeled probability. The step of performing loss fusion on the first output loss value and the second output loss value to obtain the output loss value includes: The first output loss value is weighted using the first labeled probability to obtain the first weighted loss value. The second output loss value is weighted using the second labeled probability to obtain the second weighted loss value. The output loss value is obtained based on the first weighted loss value and the second weighted loss value.

5. The model training method as described in claim 1, characterized in that, The network layer loss value includes a feature extraction loss value, the first output result includes a first feature, and the second output result includes a second feature; The step of calculating the loss of the target network layer based on the first output result and the second output result to obtain the network layer loss value includes: When the dimensions of the first feature and the second feature are inconsistent, a spatial transformation is performed on the first feature to obtain the transformed first feature; Calculate the eigennorm 2 between the eigenvalues ​​of the second feature and the eigenvalues ​​of the transformed first feature; The feature extraction loss value is determined based on the feature L2 norm.

6. The model training method as described in claim 1, characterized in that, The network layer loss value includes an attention loss value, the first output result includes a first attention probability distribution, and the second output result includes a second attention probability distribution; The step of calculating the loss of the target network layer based on the first output result and the second output result to obtain the network layer loss value includes: Calculate the relative attention entropy between the first attention probability distribution and the second attention probability distribution; The attention loss value is determined based on the relative attention entropy.

7. The model training method as described in claim 1, characterized in that, The network layer loss value includes a classification loss value, the first output result includes a first classification probability distribution, and the second output result includes a second classification probability distribution. The step of calculating the loss of the target network layer based on the first output result and the second output result to obtain the network layer loss value includes: Calculate the relative entropy between the first classification probability distribution and the second classification probability distribution; The classification loss value is determined based on the relative entropy of the classification.

8. The model training method as described in claim 1, characterized in that, The pre-trained model includes multiple network modules, and the target model includes multiple network layers, with each network module corresponding to one of the network layers.

9. The model training method as described in claim 1, characterized in that, The acquisition of the pre-trained model and the target model includes: Obtain a pre-trained model and a target model to be processed, wherein the target model to be processed includes multiple network layers to be processed; The pre-trained model is divided into blocks to obtain multiple network modules, and each network module corresponds one-to-one with the network layer to be processed. Obtain the parameters of the nth network layer in the network module, where n is a positive integer; The parameters of the nth network layer are used as the initial parameters of the corresponding network layer to be processed to obtain the target model.

10. The model training method as described in claim 9, characterized in that, After obtaining the parameters of the nth network layer in the network module, the process further includes: When the dimensions of the nth network layer are inconsistent with those of the network layer to be processed, the parameters of the network layer to be processed are randomly initialized to obtain a randomly initialized network layer. The randomly initialized network layer is trained with a mask to obtain a mask-trained network layer, and the target model includes the mask-trained network layer.

11. The model training method as described in claim 1, characterized in that, The step of linearly combining the unlabeled sample group and the labeled sample group based on the unlabeled probability and the labeled probability respectively to obtain the unlabeled amplified sample corresponding to the unlabeled sample group and the labeled amplified sample corresponding to the labeled sample group includes: Based on the unlabeled probability, the unlabeled sample group is linearly combined to obtain the unlabeled amplified sample corresponding to the unlabeled sample group; Based on the annotation probability, the annotated sample group is linearly combined to obtain the annotated augmented sample corresponding to the annotated sample group; The annotation probabilities include a first annotation probability and a second annotation probability, the annotation sample set includes a first annotation sample and a second annotation sample, and the step of linearly combining the annotation sample set based on the annotation probabilities to obtain the annotation amplification sample corresponding to the annotation sample set includes: The first labeled samples are weighted using the first labeled probability to obtain the first sample matrix; The second labeled samples are weighted using the second labeling probability to obtain the second sample matrix; The first sample matrix and the second sample matrix are fused to obtain the labeled amplified samples.

12. A model training device, characterized in that, include: The first acquisition unit is used to acquire the pre-trained model and the target model; The second acquisition unit is used to acquire an unlabeled sample group and a labeled sample group, as well as an unlabeled probability and a labeled probability. The unlabeled sample group includes at least two unlabeled samples, and the labeled sample group includes at least two labeled samples. An amplification unit is configured to linearly combine the unlabeled sample group and the labeled sample group based on the unlabeled probability and the labeled probability, respectively, to obtain unlabeled amplified samples corresponding to the unlabeled sample group and labeled amplified samples corresponding to the labeled sample group; wherein, the labeled amplified samples include tags; The preliminary training unit is used to perform preliminary training on the target model based on the unlabeled amplified samples and the pre-trained model to obtain the pre-trained target model. The training includes: inputting the unlabeled amplified samples into the pre-trained model to obtain a third output result of the target network module, where the target network module is any one of multiple network modules; inputting the unlabeled amplified samples into the target model to obtain a fourth output result of the target network layer, where the target network layer is the network layer corresponding to the target network module among the multiple network layers; calculating the loss of the target network layer based on the third and fourth output results to obtain a network layer loss value; and updating the parameters from the input layer to the target network layer in the target model based on the network layer loss value to obtain the pre-trained target model. The retraining unit, used to retrain the initially trained target model based on the labeled augmented samples and the pre-trained model, to obtain a trained target model, includes: inputting the labeled augmented samples into the pre-trained model to obtain a first output result of the target network module, wherein the target network module is any one of the plurality of network modules; inputting the labeled augmented samples into the initially trained target model to obtain a second output result of the target network layer, wherein the target network layer is the network layer corresponding to the target network module among the plurality of network layers; calculating the loss of the target network layer based on the first output result and the second output result to obtain a network layer loss value; updating the parameters from the input layer to the target network layer in the target model based on the network layer loss value; calculating the loss of the target model based on the second output result and the labels to obtain an output loss value; and updating the parameters of the target model again based on the output loss value to obtain a trained target model.

13. The model training apparatus as described in claim 12, characterized in that, The pre-trained model includes multiple network modules, and the target model includes multiple network layers, with each network module corresponding to a network layer.

14. The model training apparatus as described in claim 12, characterized in that, The labels include a first label and a second label, and the retraining unit is specifically used for: Based on the first label and the second output result, the loss of the target model is calculated to obtain the first output loss value; Based on the second label and the second output result, the loss of the target model is calculated to obtain the second output loss value; The first output loss value and the second output loss value are fused to obtain the output loss value.

15. The model training apparatus as described in claim 14, characterized in that, The annotation probabilities include a first annotation probability and a second annotation probability. The retraining unit is specifically used for: The first output loss value is weighted using the first labeled probability to obtain the first weighted loss value. The second output loss value is weighted using the second labeled probability to obtain the second weighted loss value. The output loss value is obtained based on the first weighted loss value and the second weighted loss value.

16. The model training apparatus as described in claim 12, characterized in that, The network layer loss value includes the feature extraction loss value; the first output includes the first feature; the second output includes the second feature; the retraining unit is specifically used for: When the dimensions of the first feature and the second feature are inconsistent, a spatial transformation is performed on the first feature to obtain the transformed first feature; Calculate the eigennorm 2 between the eigenvalues ​​of the second feature and the eigenvalues ​​of the transformed first feature; The feature extraction loss value is determined based on the feature L2 norm.

17. The model training apparatus as described in claim 12, characterized in that, The network layer loss value includes the attention loss value; the first output includes the first attention probability distribution; the second output includes the second attention probability distribution; the retraining unit is specifically used for: Calculate the relative attention entropy between the first attention probability distribution and the second attention probability distribution; The attention loss value is determined based on the relative entropy of attention.

18. The model training apparatus as described in claim 12, characterized in that, The network layer loss value includes a classification loss value; the first output includes a first classification probability distribution; and the second output includes a second classification probability distribution. The retraining unit is specifically used for: Calculate the relative entropy between the first and second classification probability distributions; The classification loss value is determined based on the relative entropy of classification.

19. The model training apparatus as described in claim 12, characterized in that, The pre-trained model consists of multiple network modules, and the target model consists of multiple network layers. There is a one-to-one correspondence between network modules and network layers.

20. The model training apparatus as described in claim 12, characterized in that, The first acquisition unit is specifically used for: Obtain the pre-trained model and the target model to be processed, which includes multiple network layers to be processed; The pre-trained model is divided into blocks to obtain multiple network modules, and each network module corresponds one-to-one with the network layer to be processed. Obtain the parameters of the nth network layer in the network module, where n is a positive integer; The parameters of the nth network layer are used as the initial parameters of the corresponding network layer to be processed to obtain the target model.

21. The model training apparatus as described in claim 20, characterized in that, The first acquisition unit is specifically used for: When the dimensions of the nth network layer are inconsistent with those of the network layer to be processed, the parameters of the network layer to be processed are randomly initialized to obtain the randomly initialized network layer. The randomly initialized network layers are trained with masks to obtain masked network layers. The target model includes the masked network layers.

22. The model training apparatus as described in claim 12, characterized in that, The amplification unit is specifically used for: Based on the unlabeled probability, the unlabeled sample groups are linearly combined to obtain the unlabeled amplified samples corresponding to the unlabeled sample groups. Based on the annotation probability, the annotated sample groups are linearly combined to obtain the annotated augmented samples corresponding to the annotated sample groups; The annotation probabilities include a first annotation probability and a second annotation probability. The annotation sample set includes a first annotation sample and a second annotation sample. Based on the annotation probabilities, the annotation sample set is linearly combined to obtain the corresponding annotation augmentation samples, including: The first labeled samples are weighted using the first labeled probability to obtain the first sample matrix; The second labeled samples are weighted using the second labeled probability to obtain the second sample matrix; The first sample matrix and the second sample matrix are fused to obtain the labeled and amplified samples.

23. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the model training method as described in any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the steps of the model training method according to any one of claims 1 to 11.

25. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the model training method according to any one of claims 1 to 11.

Images