Text processing model distillation method and device, computer device and medium

Abstract

The application discloses a text processing model distillation method, which comprises the following steps: obtaining original data and original labels corresponding to the original data, training a first preset model to obtain a first prediction model; predicting the original data through the first prediction model to obtain prediction labels, and dividing all the prediction labels into target labels and substandard labels; training a second preset model according to the substandard labels and original data corresponding to the substandard labels to obtain a second prediction model, and optimizing the substandard labels to obtain optimized labels; and performing distillation learning on a third preset model through the original data, the original labels, the target labels and the optimized labels to obtain a text processing model. The prediction labels of the first prediction model and the second prediction model are distilled into the text processing model through one-time distillation, thereby improving the prediction accuracy of the text processing model and improving the efficiency of the text processing model distillation.

Description

Technical Field

[0001] This invention relates to the field of predictive model technology, and in particular to a text processing model distillation method, apparatus, computer equipment, and medium. Background Technology

[0002] With the development of science and technology, natural language processing (NLP) technology has been gradually applied to various fields. Examples include keyword extraction, entity recognition, and phrase extraction. These technologies often require training models for implementation. For instance, training a text processing model to extract keywords from text.

[0003] In existing technologies, text processing models are often trained using labeled text data. For text data with poor prediction performance, the model's parameters often need to be adjusted. However, this can lead to the adjusted model failing to maintain its predictive ability for text data that performed well in the previous round. Consequently, the trained text processing model has a low accuracy rate. Summary of the Invention

[0004] This invention provides a text processing model distillation method, apparatus, computer equipment, and medium to address the problems of low accuracy in model prediction and low model training efficiency in the prior art.

[0005] A text processing model distillation method, comprising:

[0006] Obtain the raw data and the raw labels corresponding to the raw data, and train the first preset model based on the raw data and the raw labels to obtain the first prediction model;

[0007] The original data is predicted using the first prediction model to obtain predicted labels, and all predicted labels are divided into target labels and non-target labels.

[0008] The second preset model is trained based on the non-compliant labels and the original data corresponding to the non-compliant labels to obtain a second prediction model, and the non-compliant labels are optimized using the second prediction model to obtain optimized labels;

[0009] A text processing model is obtained by distilling the third preset model using the original data, the original labels, the target labels, and the optimized labels.

[0010] A text processing model distillation apparatus, comprising:

[0011] The prediction module is used to acquire raw data and the raw labels corresponding to the raw data, and to train a first preset model based on the raw data and the raw labels to obtain a first prediction model.

[0012] The classification module is used to predict the original data using the first prediction model, obtain predicted labels, and classify all predicted labels into target labels and non-target labels.

[0013] An optimization module is used to train a second preset model based on the non-compliant labels and the original data corresponding to the non-compliant labels to obtain a second prediction model, and to optimize the non-compliant labels using the second prediction model to obtain optimized labels;

[0014] The distillation module is used to perform distillation learning on the third preset model using the original data, the original labels, the target labels, and the optimized labels to obtain a text processing model.

[0015] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described text processing model distillation method.

[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described text processing model distillation method.

[0017] This invention provides a text processing model distillation method, apparatus, computer device, and storage medium. The method uses a trained first prediction model to predict raw data, enabling the first prediction model to learn features from labels with good prediction performance (i.e., target labels). Labels with poor prediction performance from the first prediction model (i.e., substandard labels) are then fed to a second prediction model for learning. In this way, two different models can learn different data features from the raw data, improving the efficiency and accuracy of model training. Furthermore, by distilling the target labels predicted well by the first prediction model and the optimized labels predicted well by the second prediction model into a third preset model, the third preset model can learn the predictive advantages of both the first and second prediction models, thereby improving the training efficiency and accuracy of the text processing model. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram of the application environment of the text processing model distillation method in one embodiment of the present invention;

[0020] Figure 2 This is a flowchart of a text processing model distillation method in one embodiment of the present invention;

[0021] Figure 3 This is a flowchart of step S20 in the text processing model distillation method of one embodiment of the present invention;

[0022] Figure 4 This is a flowchart of step S40 in the text processing model distillation method of an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of a text processing model distillation apparatus according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The text processing model distillation method provided in this embodiment of the invention can be applied to, for example... Figure 1 The application environment is shown. Specifically, this text processing model distillation method is applied in a text processing model distillation apparatus, which includes, for example, [details of the apparatus]. Figure 1 The client and server shown communicate over a network to address the low accuracy and efficiency issues of existing text processing models. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The client, also known as the user terminal, refers to the program that provides local services to the client, corresponding to the server. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0027] In one embodiment, such as Figure 2As shown, a text processing model distillation method is provided, which is applied to... Figure 1 Taking the server in the example, the following steps are included:

[0028] S10: Obtain the original data and the original labels corresponding to the original data, and train the first preset model based on the original data and the original labels to obtain the first prediction model.

[0029] Understandably, the raw data can be collected from different websites or databases using web scraping techniques. In this embodiment, the raw data is text data (text data can be Chinese text, English text, or text containing both Chinese and English). The raw tag serves as a representation of the raw data, and its meaning varies depending on the application scenario. For example, in a keyword extraction application scenario, the raw tag represents the keywords in the raw data. In this case, keywords can be extracted from the raw data as the raw tag through manual annotation or keyword recognition. In an entity recognition application scenario, the raw tag can be the entity meaning of a specific character in the raw data. In this case, the entity meaning corresponding to different words in the raw data can be determined through manual annotation or entity recognition as the raw tag.

[0030] Furthermore, the first preset model is a model built based on the BERT model, used for label prediction on the original data. The first preset model can be a BERT-Seq2Seq model, which includes multiple encoder layers and can recognize text of different lengths. The first prediction model is obtained by training the first preset model using the original data.

[0031] Specifically, the original data and its corresponding original labels are retrieved from the server's database and input into a first preset model. The first preset model is then trained using the original data. That is, the initial parameters of the first preset model are adjusted based on the original labels and the model's prediction results, causing the label prediction results of the adjusted first preset model to continuously converge towards the original labels. When the predicted labels of the first preset model based on the adjusted initial parameters meet the convergence condition, training ends, and the converged first preset model is determined as the first prediction model.

[0032] S20: The original data is predicted using the first prediction model to obtain predicted labels, and all predicted labels are divided into target labels and non-target labels.

[0033] Understandably, the predicted label is the result of the first prediction model's label prediction on the original data. The target label is the predicted label that is greater than or equal to a preset label threshold. The unsuccessful label is the predicted label that is less than the preset label threshold. The preset label threshold is used to determine whether the original label and the predicted label are similar. The preset label threshold can be an F1 score, which can be set to 0.9. The F1 score is a statistical indicator used to measure model accuracy. That is, the F1 score of the predicted label is calculated and compared with the preset label threshold. The preset label threshold can also be Euclidean distance or cosine similarity, which can be set to 0.95. That is, the Euclidean distance or cosine similarity between the original label and the predicted label is calculated and compared with the preset label threshold.

[0034] Specifically, after obtaining the first prediction model, the original data is input into it. The embedding layer in the first prediction model performs vector transformation on the original data, converting it into vectors to obtain the corresponding embedding vectors. This is achieved through multiple sets of three weight matrices W in the attention layer. Q W K W V The embedding vectors are calculated to obtain the corresponding Query vector, Keys vector, and Values ​​vector. The relevance score between the embedding vectors is calculated using the dot product method, i.e., the dot product is calculated between each embedding vector in Q and each embedding vector in K. The relevance score between the embedding vectors is normalized by using the softmax function to convert the score into a probability distribution between [0, 1]. Based on the probability distribution between the embedding vectors, the corresponding Values ​​value is multiplied to obtain a matrix. Multiple matrices are concatenated, and the concatenated matrix is ​​processed through a first residual connection layer to avoid degradation during model training. The processed matrix is ​​then normalized again through a first normalization layer. Finally, the normalized matrix is ​​activated using the ReLU function, and the activated matrix is ​​normalized again through a second residual connection layer and a second normalization layer to obtain the predicted label.

[0035] Furthermore, loss calculations are performed on the original labels and predicted labels corresponding to the same original data. This can be done by calculating the difference between the original and predicted labels using a loss function, directly obtaining the predicted value for each predicted label. Alternatively, the Euclidean distance or cosine similarity between the original and predicted labels can be calculated first, and the predicted value for each predicted label can be determined based on this Euclidean distance or cosine similarity. In this way, the predicted values ​​for each predicted label can be determined using the above methods, and all predicted labels can be categorized into target labels and non-target labels based on the predicted values ​​and a preset label threshold. The loss function can be the CTC loss function, the Focal Loss loss function, etc.

[0036] S30: Train the second preset model based on the non-compliant label and the original data corresponding to the non-compliant label to obtain the second prediction model, and optimize the non-compliant label using the second prediction model to obtain the optimized label.

[0037] Understandably, the second preset model is a model built based on the BERT model, used to predict labels for the original data corresponding to the non-compliant labels. The second preset model can be a BERT-Seq2Seq model or a BERT-Dense model. Further, the first and second preset models can be the same model or different models. When the two models are the same, they can both be BERT models; when the two models are different, the first preset model can be a BERT-Seq2Seq model, and the second preset model can be a BERT-Dense model. In this embodiment, the first and second preset models are preferably different models. Because when the original data corresponding to the non-compliant labels, whose prediction performance is poor after training the first preset model (i.e., the first prediction model), is input again into the same model (i.e., the second preset model) for training, the second preset model still cannot learn other features of the original data corresponding to the non-compliant labels. This results in a poor learning performance of the second preset model, which in turn affects the distillation learning of the third preset model in subsequent steps. The second prediction model is obtained by training the second preset model using the original data corresponding to the non-compliant labels. The optimized label is the label obtained by optimizing the unqualified label using the second prediction model.

[0038] Specifically, after obtaining the unqualified labels, the original data corresponding to the unqualified labels is acquired and input into the second preset model for training. That is, the initial parameters of the second preset model are adjusted using the prediction results of the original labels and the second preset model, so that the label prediction results of the adjusted second preset model continuously approach the original labels. When the predicted label value of the second preset model for the original data reaches the convergence condition, training ends, and the converged second preset model is determined as the second prediction model. The second prediction model is then used to re-predict the original data corresponding to the unqualified labels, and the predicted label of the second prediction model replaces the unqualified labels. This predicted label is then determined as the optimized label, thus obtaining the optimized label.

[0039] S40: Distillation learning is performed on the third preset model using the original data, the original label, the target label, and the optimized label to obtain a text processing model.

[0040] Understandably, the third preset model can be the TextCNN model, used for distillation learning of the first and second prediction models. The text processing model is a model that acquires the predictive capabilities of the first and second prediction models through distillation learning.

[0041] Specifically, after obtaining the optimized labels, the third preset model is distilled using the original data and original labels. The distilled third preset model then predicts the original data to obtain the model prediction results. The target label and optimized label are distilled into the distilled third preset model, and the prediction results of the target label or optimized label corresponding to the same original data are compared with those of the third preset model to determine the loss value of the third preset model. The initial parameters of the distilled third preset model are adjusted based on its loss value. When the loss value of the third preset model reaches the convergence condition, the distilled third preset model is determined as the text processing model.

[0042] In this embodiment of the invention, the method uses a trained first prediction model to predict the original data, thereby enabling the first prediction model to learn the features of labels with better prediction performance (i.e., target labels). Labels with poor prediction performance from the first prediction model (i.e., substandard labels) are then fed to a second prediction model for learning. In this way, two different models can learn different data features from the original data, improving the efficiency and accuracy of model training. Furthermore, by distilling the target labels predicted well by the first prediction model and the optimized labels predicted well by the second prediction model into a third preset model, the third preset model can learn the predictive advantages of both the first and second prediction models, thereby improving the training efficiency and accuracy of the text processing model.

[0043] In one embodiment, step S10, namely training the first preset model based on the original data and the original labels to obtain the first prediction model, includes:

[0044] S101, The original data is input into the first preset model, and the original data is predicted by the first preset model to obtain the second training label.

[0045] Understandably, the second training label is the model prediction result obtained by the first preset model predicting the original data.

[0046] Specifically, after obtaining the original data and original labels, the original data is input into the first preset model. The embedding layer in the first preset model transforms the original data to obtain the embedding vectors corresponding to the original data. The attention layer calculates the embedding vectors to obtain the matrix corresponding to the embedding vectors. Multiple matrices are concatenated, and the concatenated matrix is ​​processed through the first residual connection layer to avoid degradation problems during model training. Then, the processed matrix is ​​normalized through the first normalization layer. Then, the normalized matrix is ​​activated by the ReLU function, and the activated matrix is ​​normalized through the second residual connection layer and the second normalization layer to obtain the second training label. The specific process is the same as step S20 above, and will not be repeated here. Among them, the accuracy of the prediction of the first prediction model is much greater than that of the first preset model. This embodiment only lists one encoder layer in the first preset model. The first preset model includes multiple encoder layers. The second training label predicted by the first preset model can be obtained through all the encoder layers in the first preset model.

[0047] S102, determine the third loss value based on the second training label and the original label.

[0048] Understandably, the third loss value is the loss value of the first preset model, that is, the loss generated during the training of the first preset model.

[0049] Specifically, after obtaining the second training label, the difference between the second training label and the original label is calculated. This difference can be calculated using the CTC loss function or the Focal Loss function to determine the loss value of the first preset model, thus obtaining the third loss value. Alternatively, the Euclidean distance or cosine similarity between the second training label and the original label can be calculated first, and the third loss value can be determined based on this Euclidean distance or cosine similarity.

[0050] S103, optimize the first preset model according to the third loss value to obtain the first prediction model.

[0051] Specifically, after obtaining the third loss value, the initial parameters of each layer in the first preset model are optimized based on the third loss value. The optimized first preset model is then used to predict the original data to obtain the corresponding second training label. The loss is calculated using the CTC loss function on the new second training label and the target label or optimized label corresponding to the same original data to obtain a new third loss value. It is then determined whether the new third loss value meets the convergence condition. If the new third loss value meets the convergence condition, the optimized first preset model is determined as the first prediction model. If the new third loss value does not meet the convergence condition, the initial parameters of each layer in the first preset model are re-optimized based on the new third loss value. This process continues until the loss value of the first preset model meets the convergence condition, at which point the first preset model that meets the convergence condition is recorded as the first prediction model.

[0052] In this embodiment of the invention, a first preset model is used to predict the original data to obtain a second training label. Based on the second training label and the original label, a third loss value is determined. The first preset model is then optimized using the third loss value until it meets the convergence condition, thus determining the first prediction model and improving the accuracy of predictions made by the first prediction model.

[0053] In one embodiment, such as Figure 3 As shown, in step S20, dividing all the predicted labels into target labels and non-target labels includes:

[0054] S201: Based on the original label and the predicted label corresponding to the same original data, determine the predicted value corresponding to the predicted label.

[0055] Understandably, the predicted value is used to characterize the similarity between the original label and the predicted label.

[0056] Specifically, after obtaining the target label and the labels that did not meet the target, the original label and the predicted label corresponding to the same original data are obtained. The difference between the original label and the predicted label corresponding to the same original data is calculated using the CTC loss function or the Focal Loss function, and the result is determined as the predicted value corresponding to the predicted label. In this way, the predicted value corresponding to each predicted label can be obtained.

[0057] S202: Obtain a preset label threshold and compare the predicted value with the preset label threshold.

[0058] S203: The predicted label corresponding to the predicted value that is greater than or equal to the preset label threshold is determined as the target label, and the predicted label corresponding to the predicted value that is less than the preset label threshold is determined as the unqualified label.

[0059] Understandably, the preset label threshold is used to determine the similarity between the original label and the predicted label. The target label is the predicted label corresponding to the predicted value that is greater than or equal to the preset label threshold. The unqualified label is the predicted label corresponding to the predicted value that is less than the preset label threshold.

[0060] Specifically, after obtaining the predicted value corresponding to the predicted label, a preset label threshold is retrieved from the server or obtained from a third-party platform, and the predicted value corresponding to the predicted label is compared with the preset label threshold. When the predicted value corresponding to the predicted label is greater than or equal to the preset label threshold, the predicted label corresponding to the predicted value greater than or equal to the preset label threshold is determined as the target label. When the predicted value corresponding to the predicted label is less than the preset label threshold, the predicted label corresponding to the predicted value less than the preset label threshold is determined as the unqualified label. In this way, all target labels and all unqualified labels can be obtained.

[0061] This invention, through the use of original labels and predicted labels corresponding to the same original data, determines the predicted value corresponding to the predicted label. By comparing a preset label threshold with all predicted values, the target label and the labels that do not meet the threshold are determined, facilitating the subsequent training of the second preset model.

[0062] In one embodiment, step S30, namely training the second preset model based on the non-compliance labels and the original data corresponding to the non-compliance labels to obtain the second prediction model, includes:

[0063] S301, the original data corresponding to the unqualified label is input into the second preset model, and the second preset model is used to predict the original data corresponding to the unqualified label to obtain the third training label.

[0064] Understandably, the third training label is the model prediction result obtained by the second preset model from the original data corresponding to the label that did not meet the standard.

[0065] Specifically, after obtaining the target label and the non-target labels, the original data corresponding to the non-target labels is obtained through the non-target labels. This original data is then input into the second preset model, which predicts the original data corresponding to the non-target labels. Specifically, when the second preset model is a Bert-Dense model, the embedding layer in the second preset model performs vector transformation on the original data corresponding to the non-target labels to obtain the embedding vectors. The specific process is the same as step S20 above, and will not be repeated here; only the differences will be explained. That is, after activating the normalized matrix using the ReLU function, the activated matrix is ​​input into the fully connected layer. The hidden layers in the fully connected layer process all activated matrices using different weights. The processing result is then input into the second residual connected layer through the output layer of the fully connected layer. The second residual connected layer and the second normalization layer then predict the processing result to obtain the third training label.

[0066] S302, determine the fourth loss value based on the third training label and the unqualified label corresponding to the same original data.

[0067] Understandably, the fourth loss value is the loss value of the second preset model, that is, the loss generated during the training of the second preset model.

[0068] Specifically, after obtaining the third training label, the difference between the third training label and the original label is calculated using the CTC loss function to determine the loss value of the second preset model, thus obtaining the fourth loss value. Alternatively, the Euclidean distance or cosine similarity between the third training label and the original label can be calculated first, and the fourth loss value can be determined based on this Euclidean distance or cosine similarity.

[0069] S303, the second preset model is optimized using the fourth loss value to obtain the second prediction model.

[0070] Specifically, after obtaining the fourth loss value, the initial parameters of each layer in the second preset model are optimized based on the fourth loss value. The optimized second preset model is then used to predict the original data corresponding to the unqualified labels to obtain the corresponding third training labels. The loss is calculated using the CTC loss function on the new third training labels and the target or optimized labels corresponding to the same original data to obtain a new fourth loss value. It is then determined whether the new fourth loss value meets the convergence condition. If the new fourth loss value meets the convergence condition, the optimized second preset model is determined as the second prediction model. If the new fourth loss value does not meet the convergence condition, the initial parameters of each layer in the second preset model are re-optimized based on the new fourth loss value. This process continues until the loss value of the second preset model meets the convergence condition, at which point the convergent second preset model is recorded as the second prediction model.

[0071] This invention trains a second preset model using the original data corresponding to the non-compliant labels, reducing the number of parameters and improving model training efficiency. The second preset model predicts the original data corresponding to the non-compliant labels to obtain a third training label, and based on the third training label and the original labels, a second loss value is determined. The second preset model is then optimized using the second loss value until it meets the convergence condition, thus determining the second prediction model and improving the accuracy of predictions made by the second prediction model.

[0072] In one embodiment, such as Figure 4 As shown, in step S40, the third preset model is distilled and learned using the original data, the original labels, the target labels, and the optimized labels to obtain a text processing model, including:

[0073] S401, Input the original data and the original labels into the third preset model, and perform distillation learning on the third preset model using the original data and the original labels to obtain the distillation model.

[0074] S402, the original data is predicted using the distillation model to obtain the first training label.

[0075] Understandably, the third preset model is the TextCNN model, which includes embedding layers, convolutional layers, pooling layers, and fully connected layers. The distillation model is obtained by distilling the third preset model using the original data and original labels. The first training label is obtained by the distillation model predicting the original data.

[0076] Specifically, after obtaining the optimized labels, the original data and original labels are input into the third preset model. The third preset model undergoes distillation learning using the original data and original labels, enabling it to have basic predictive capabilities. This distilled model is then designated as the distillation model. Predictions are made based on the distillation model using the original data. First, the original data is vectorized through the embedding layer in the distillation model to obtain the corresponding embedding vectors. Then, a convolutional layer performs one-dimensional convolution on the embedding vectors to obtain convolutional feature vectors. Next, a pooling layer performs max pooling on the convolutional feature vectors, transforming convolutional feature vectors of different lengths into fixed-length vectors and concatenating them into a pooled text vector. Finally, a fully connected layer processes the pooled text vector to avoid overfitting, and the processed pooled text vector is used for prediction to obtain the first training label.

[0077] S403, the target label and the optimized label are distilled into the distillation model, and a first loss value is determined based on the target label, the optimized label and the first training label.

[0078] S404, optimize the distillation model based on the first loss value, and determine whether the first loss value meets the convergence condition. When the first loss value meets the convergence condition, determine the distillation model as the text processing model.

[0079] Understandably, the first loss value is the loss value of the third preset model, that is, the loss generated during the training of the third preset model.

[0080] Specifically, after obtaining the first training label, the target label and optimized label are distilled into the distillation model through distillation. This means that the predictive advantages of the first and second prediction models are distilled into the distillation model, thus giving it the predictive capabilities of both models. Then, loss is calculated using the first training label and the target or optimized label corresponding to the same original data. Specifically, the difference between the first training label and the target or optimized label is calculated using the CTC loss function to obtain the first loss value. Based on the first loss value, the initial parameters of each layer in the distillation model are optimized to obtain an optimized distillation model. The optimized distillation model then predicts the original data to obtain a new first training label. The loss is then calculated using the CTC loss function on the new first training label and the target or optimized label corresponding to the same original data to obtain a new first loss value. Finally, it is determined whether the new first loss value meets the convergence condition. If the new first loss value meets the convergence condition, the new distillation model is identified as the text processing model.

[0081] This invention distills the target label and optimization label into a third preset model, thereby distilling the model structures of the first and second prediction models into the third preset model. It then determines whether the first loss value of the third preset model meets the convergence condition, thus establishing the text processing model. This improves the accuracy of the text processing model's prediction of the original data and increases the efficiency of text processing model distillation.

[0082] In one embodiment, step S404, which involves optimizing the distillation model based on the first loss value and determining whether the first loss value meets the convergence condition, includes:

[0083] S4041, if the first loss value does not meet the convergence condition, the initial parameters of the distillation model are adjusted to obtain the target distillation model.

[0084] S4042, the original data is predicted using the target distillation model to obtain a distillation label; a second loss value is determined based on the distillation label and the original label.

[0085] Understandably, the target distillation model is obtained by adjusting the initial parameters of the distillation model using the first loss value. The second loss value is the loss value of the target distillation model, which is the loss generated during the training of the distillation model.

[0086] Specifically, when the first loss value does not meet the convergence condition, the initial parameters of each layer in the distillation model are readjusted based on the first loss value, and the distillation model with adjusted initial parameters is determined as the target distillation model. The original data is then predicted using the target distillation model to obtain the distillation label corresponding to the original data. The second loss value can be obtained by calculating the loss between the distillation label and the target label or optimized label corresponding to the same original data using the CTC loss function.

[0087] S4043, if the second loss value does not reach the convergence condition, iteratively update the initial parameters in the target distillation model until the second loss value reaches the convergence condition, and record the converged target distillation model as the text processing model.

[0088] Understandably, the convergence condition can be that the second loss value is less than a set threshold, that is, when the second loss value is less than the set threshold, training stops; the convergence condition can also be that the second loss value is very small and will not decrease further after 500 calculations, that is, when the second loss value is very small and will not decrease further after 500 calculations, training stops.

[0089] Specifically, after determining the second loss value, if the second loss value does not reach the preset convergence condition, the initial parameters of the target distillation model are adjusted according to the second loss value, and the original data is re-input into the target distillation model with adjusted initial parameters to obtain the second loss value corresponding to the target distillation model with adjusted initial parameters. When the second loss value reaches the preset convergence condition, the converged target distillation model is recorded as a text processing model. If the second loss value does not reach the preset convergence condition, the initial parameters of the target distillation model are adjusted again according to the second loss value, so that the output result of the target distillation model with adjusted initial parameters can continuously approach the accurate result, making the model's prediction accuracy higher and higher. This continues until the second loss value reaches the preset convergence condition, and the converged target distillation model is recorded as a text processing model.

[0090] This invention achieves the acquisition of a target distillation model by adjusting the initial parameters of the distillation model when the first loss value has not reached the convergence condition. The target distillation model is then used to predict the original data to obtain a second loss value. When the second loss value reaches the convergence condition, the text processing model is determined, thereby improving the accuracy of the text processing model's predictions and its distillation efficiency.

[0091] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0092] In one embodiment, a text processing model distillation apparatus is provided, which corresponds one-to-one with the text processing model distillation method described in the above embodiments. For example... Figure 5 As shown, the text processing model distillation device includes a prediction module 11, a classification module 12, an optimization module 13, and a distillation module 14. Detailed descriptions of each functional module are as follows:

[0093] Prediction module 11 is used to acquire raw data and the raw labels corresponding to the raw data, and to train a first preset model based on the raw data and the raw labels to obtain a first prediction model;

[0094] The classification module 12 is used to predict the original data through the first prediction model, obtain predicted labels, and divide all the predicted labels into target labels and non-target labels.

[0095] Optimization module 13 is used to train a second preset model based on the unqualified label and the original data corresponding to the unqualified label to obtain a second prediction model, and to optimize the unqualified label through the second prediction model to obtain an optimized label;

[0096] Distillation module 14 is used to perform distillation learning on the third preset model using the original data, the original label, the target label, and the optimized label to obtain a text processing model.

[0097] In one embodiment, the prediction module 11 includes:

[0098] The second label prediction unit is used to input the original data into the first preset model, and predict the original data through the first preset model to obtain the second training label;

[0099] The third loss value unit is used to determine the third loss value based on the second training label and the original label corresponding to the same original data;

[0100] The first prediction model unit is used to optimize the first preset model based on the third loss value to obtain the first prediction model.

[0101] In one embodiment, the classification module 12 includes:

[0102] The determining unit is configured to determine the predicted value corresponding to the predicted label based on the original label and the predicted label corresponding to the same original data;

[0103] A comparison unit is used to obtain a preset label threshold and compare the predicted value with the preset label threshold.

[0104] The result unit is used to determine the predicted label corresponding to the predicted value that is greater than or equal to the preset label threshold as the target label, and to determine the predicted label corresponding to the predicted value that is less than the preset label threshold as the unqualified label.

[0105] In one embodiment, the optimization module 13 includes:

[0106] The third label prediction unit is used to input the original data corresponding to the non-compliant label into the second preset model, and predict the original data corresponding to the non-compliant label through the second preset model to obtain the third training label;

[0107] The fourth loss value unit is used to determine the fourth loss value based on the third training label and the non-compliance label corresponding to the same original data;

[0108] The second prediction model unit is used to optimize the second preset model using the fourth loss value to obtain the second prediction model.

[0109] In one embodiment, the distillation module 14 includes:

[0110] A distillation learning unit is used to input the original data and the original labels into the third preset model, and to perform distillation learning on the third preset model using the original data and the original labels to obtain a distillation model;

[0111] The first label prediction unit is used to predict the original data using the distillation model to obtain the first training label;

[0112] The first loss value unit is used to distill the target label and the optimized label into the distillation model, and determine the first loss value based on the target label, the optimized label and the first training label;

[0113] The model determination unit is used to optimize the distillation model based on the first loss value and determine whether the first loss value meets the convergence condition. When the first loss value meets the convergence condition, the distillation model is determined as the text processing model.

[0114] In one embodiment, the text processing model unit further includes:

[0115] A parameter adjustment unit is used to adjust the initial parameters of the distillation model to obtain the target distillation model if the first loss value does not reach the convergence condition.

[0116] The second loss value unit is used to predict the original data using the target distillation model to obtain a distillation label; and to determine a second loss value based on the distillation label and the original label.

[0117] The model convergence unit is used to iteratively update the initial parameters in the target distillation model when the second loss value has not reached the convergence condition, until the second loss value reaches the convergence condition, and then record the converged target distillation model as the text processing model.

[0118] Specific limitations regarding the text processing model distillation apparatus can be found in the limitations of the text processing model distillation method described above, and will not be repeated here. Each module in the aforementioned text processing model distillation apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0119] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data used in the text processing model distillation method described in the above embodiment. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a text processing model distillation method.

[0120] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the text processing model distillation method described in the above embodiment.

[0121] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the text processing model distillation method described in the above embodiment.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0124] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A text processing model distillation method, characterized in that, include: Obtain the raw data and the raw labels corresponding to the raw data, and train the first preset model based on the raw data and the raw labels to obtain the first prediction model; The original data is predicted using the first prediction model to obtain predicted labels, and all predicted labels are divided into target labels and non-target labels. The second preset model is trained based on the non-compliant labels and the original data corresponding to the non-compliant labels to obtain a second prediction model, and the non-compliant labels are optimized using the second prediction model to obtain optimized labels; A text processing model is obtained by distilling the third preset model using the original data, the original labels, the target labels, and the optimized labels. The step of performing distillation learning on the third preset model using the original data, the original labels, the target labels, and the optimized labels to obtain a text processing model includes: The original data and the original labels are input into the third preset model, and the third preset model is subjected to distillation learning using the original data and the original labels to obtain the distillation model; The original data is predicted using the distillation model to obtain the first training label; The target label and the optimized label are distilled into the distillation model, and a first loss value is determined based on the target label, the optimized label, and the first training label. The distillation model is optimized based on the first loss value, and it is determined whether the first loss value meets the convergence condition. When the first loss value meets the convergence condition, the distillation model is determined as the text processing model. After optimizing the distillation model based on the first loss value and determining whether the first loss value meets the convergence condition, the process includes: If the first loss value does not meet the convergence condition, the initial parameters of the distillation model are adjusted to obtain the target distillation model; The original data is predicted using the target distillation model to obtain distillation labels; a second loss value is determined based on the distillation labels and the original labels. If the second loss value does not reach the convergence condition, the initial parameters in the target distillation model are iteratively updated until the second loss value reaches the convergence condition. Then, the converged target distillation model is recorded as the text processing model.

2. The text processing model distillation method as described in claim 1, characterized in that, The step of dividing all predicted labels into target labels and non-target labels includes: Based on the original label and the predicted label corresponding to the same original data, determine the predicted value corresponding to the predicted label; Obtain a preset label threshold, and compare the predicted value with the preset label threshold; The predicted label corresponding to the predicted value that is greater than or equal to the preset label threshold is determined as the target label, and the predicted label corresponding to the predicted value that is less than the preset label threshold is determined as the unqualified label.

3. The text processing model distillation method as described in claim 1, characterized in that, The step of training the first preset model based on the original data and the original labels to obtain the first prediction model includes: The raw data is input into the first preset model, and the raw data is predicted by the first preset model to obtain the second training label; A third loss value is determined based on the second training label and the original label corresponding to the same original data; The first preset model is optimized based on the third loss value to obtain the first prediction model.

4. The text processing model distillation method as described in claim 1, characterized in that, The step of training the second preset model based on the non-compliance labels and the original data corresponding to the non-compliance labels to obtain the second prediction model includes: The original data corresponding to the non-compliant labels are input into the second preset model, and the second preset model is used to predict the original data corresponding to the non-compliant labels to obtain the third training label; A fourth loss value is determined based on the third training label and the non-compliant label corresponding to the same original data; The second preset model is optimized using the fourth loss value to obtain the second prediction model.

5. A text processing model distillation apparatus, characterized in that, include: The prediction module is used to acquire raw data and the raw labels corresponding to the raw data, and to train a first preset model based on the raw data and the raw labels to obtain a first prediction model. The classification module is used to predict the original data using the first prediction model, obtain predicted labels, and classify all predicted labels into target labels and non-target labels. An optimization module is used to train a second preset model based on the non-compliant labels and the original data corresponding to the non-compliant labels to obtain a second prediction model, and to optimize the non-compliant labels using the second prediction model to obtain optimized labels; The distillation module is used to perform distillation learning on the third preset model using the original data, the original labels, the target labels, and the optimized labels to obtain a text processing model; The distillation module includes: A distillation learning unit is used to input the original data and the original labels into the third preset model, and to perform distillation learning on the third preset model using the original data and the original labels to obtain a distillation model; The first label prediction unit is used to predict the original data using the distillation model to obtain the first training label; The first loss value unit is used to distill the target label and the optimized label into the distillation model, and determine the first loss value based on the target label, the optimized label and the first training label; The model determination unit is used to optimize the distillation model based on the first loss value and determine whether the first loss value meets the convergence condition. When the first loss value meets the convergence condition, the distillation model is determined as the text processing model. In one embodiment, the text processing model further includes: A parameter adjustment unit is used to adjust the initial parameters of the distillation model to obtain the target distillation model if the first loss value does not reach the convergence condition. The second loss value unit is used to predict the original data using the target distillation model to obtain a distillation label; and to determine a second loss value based on the distillation label and the original label. The model convergence unit is used to iteratively update the initial parameters in the target distillation model when the second loss value has not reached the convergence condition, until the second loss value reaches the convergence condition, and then record the converged target distillation model as the text processing model.

6. The text processing model distillation apparatus as described in claim 5, characterized in that, The classification module includes: The determining unit is configured to determine the predicted value corresponding to the predicted label based on the original label and the predicted label corresponding to the same original data; A comparison unit is used to obtain a preset label threshold and compare the predicted value with the preset label threshold. The result unit is used to determine the predicted label corresponding to the predicted value that is greater than or equal to the preset label threshold as the target label, and to determine the predicted label corresponding to the predicted value that is less than the preset label threshold as the unqualified label.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the text processing model distillation method as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the text processing model distillation method as described in any one of claims 1 to 4.

Images