Classification model training and text classification method and device, and electronic equipment
Through multiple rounds of knowledge distillation and iterative training methods, using soft labels and parameter prediction models, the training efficiency and accuracy of text classification models are improved, and the problems of low training efficiency and overfitting in the existing technology are solved, and efficient text is achieved Classification.
Patent Information
- Application Number
- CN202310512241.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-07
- Publication Date
- 2024-11-08
AI Technical Summary
In the prior art, the training efficiency of text classification models is low and easy to overfit, resulting in low classification accuracy, especially due to the complex process of requiring a large amount of pre-labeled training data and fine-tuning specific network layers.
Multi-round knowledge distillation training and iterative training methods are adopted to adjust the initial classification model by referring to the soft labels output by the classification model, obtain the intermediate classification model, and further adjust the intermediate classification model using the parameter prediction model to improve text classification performance.
With a small amount of labeled data and training data, the model training time is quickly shortened, the text classification efficiency and accuracy are improved, the model parameters are reduced, and the classification ability is enhanced.
Smart Images

Figure CN118916480A_ABST
Abstract
Description
Background Art
[0002] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It mainly studies various theories and methods that can enable effective communication between humans and computers using natural language.
[0003] Text classification is an important task in natural language processing, involving news classification, question-answer matching, intent recognition, inference, etc. With the development of deep learning technology, how to achieve text classification by training neural network models has become a hot issue that technicians are constantly exploring.
[0004] In related technologies, in order to train a text classification model to achieve the target task, a pre-trained model trained on a large-scale corpus is first obtained, and then a specific network layer of the pre-trained model is fine-tuned through the constructed sample text set to meet the requirements of the target task. However, the above method requires obtaining a large amount of training data for a specific task, and the training data for text classification tasks basically needs to be pre-labeled, which is time-consuming and has low model training efficiency. In addition, fine-tuning a specific network layer can easily cause the model to overfit, reducing the classification accuracy of the text classification model.
[0005] Therefore, how to improve the training efficiency of text classification models and improve the accuracy of text classification has become a problem that needs to be solved urgently. Summary of the invention
[0006] The embodiments of the present application provide a classification model training and text classification method, device, and electronic device to improve the classification model training efficiency and text classification accuracy.
[0007] A classification model training method provided in an embodiment of the present application includes:
[0008] Based on the first sample text set and the reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: in one round of knowledge distillation training, output soft labels for adjusting parameters of the initial classification model, and the soft labels include: reference estimated values of the first sample text corresponding to the reference classification model belonging to each preset text category;
[0009] Based on the second sample text set and the trained parameter prediction model, the intermediate classification model is subjected to multiple rounds of iterative training to obtain a target classification model, wherein the sample texts contained in the second sample text set are not completely the same as those contained in the first sample text set; the parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting the parameters of the intermediate classification model.
[0010] A text classification method provided in an embodiment of the present application includes:
[0011] Get the text to be classified;
[0012] The text to be classified is input into a trained target classification model to obtain the target text category of the text to be classified, wherein the target classification model is obtained in the following manner:
[0013] Based on the first sample text set and the reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training;
[0014] Based on the second sample text set and the trained parameter prediction model, the intermediate classification model is subjected to multiple rounds of iterative training to obtain a target classification model. The parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting parameters of the intermediate classification model.
[0015] The present invention provides a classification model training device, comprising:
[0016] A distillation unit is used to perform multiple rounds of knowledge distillation training on the initial classification model based on the first sample text set and the reference classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training, and the soft labels include: reference estimated values of the first sample text corresponding to the reference classification model belonging to each preset text category;
[0017] A training unit is used to perform multiple rounds of iterative training on the intermediate classification model based on a second sample text set and a trained parameter prediction model to obtain a target classification model, wherein the sample texts contained in the second sample text set are not completely the same as those contained in the first sample text set; and the parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting the parameters of the intermediate classification model.
[0018] Optionally, the training unit is further used for:
[0019] Obtain a model to be trained, multiple candidate classification models, and a third sample text set of each of the multiple candidate classification models, wherein the network structures of the multiple candidate classification models are not completely the same, and the third sample text set includes a training sample text set and a verification sample text set;
[0020] Based on each training sample text set and the model to be trained, the multiple candidate classification models are respectively subjected to multiple rounds of iterative training to obtain each control classification model;
[0021] Based on the control classification models and the verification sample text sets, the model to be trained is subjected to multiple rounds of iterative training to obtain the parameter prediction model.
[0022] Optionally, each training sample text is provided with a corresponding text category label; then the training unit is specifically used for:
[0023] For the multiple candidate classification models, in one round of iterative training, the following operations are performed respectively:
[0024] Input the selected training sample text into a candidate classification model to obtain the corresponding predicted text category;
[0025] Constructing a classification loss function based on the difference between the predicted text category and the corresponding text category label;
[0026] Obtaining a classification gradient based on the classification loss function and a current model parameter of the candidate classification model;
[0027] The classification gradient is input into the model to be trained for parameter prediction to obtain an output prediction parameter set, and based on the prediction parameter set, parameters of the candidate classification model are adjusted.
[0028] Optionally, each verification sample text is provided with a corresponding text category label; then the training unit is specifically used for:
[0029] In one round of training iteration, the following operations are performed:
[0030] Inputting each verification sample text selected from each verification sample text set into a corresponding comparison classification model to obtain output each predicted text category;
[0031] Based on the difference between each predicted text category and the corresponding text category label, constructing a verification loss function for each comparison classification model;
[0032] Based on the verification loss functions, parameters of the model to be trained are adjusted.
[0033] Optionally, each second sample text is provided with a corresponding text category label; the training unit is specifically used for:
[0034] Inputting the selected second sample text into the intermediate classification model to obtain a corresponding predicted text category;
[0035] By using the parameter prediction model, a target loss function constructed based on the predicted text category and the corresponding text category label is used to perform parameter prediction to obtain an output prediction parameter set;
[0036] Based on the prediction parameter set, parameters of the intermediate classification model are adjusted.
[0037] Optionally, the training unit is specifically used for:
[0038] Obtaining a target gradient based on the target loss function and current model parameters of the intermediate classification model;
[0039] The target gradient is input into the parameter prediction model to perform parameter prediction to obtain the prediction parameter set.
[0040] Optionally, the distillation unit is specifically used for:
[0041] In one round of knowledge distillation training, the following operations are performed:
[0042] Input the selected first sample text into the reference classification model and the initial classification model respectively, and obtain the soft label output by the reference classification model and the soft label output by the initial classification model, wherein the soft label output by the initial classification model includes: the initial estimated value of each of the preset text categories to which the first sample text belongs respectively;
[0043] Based on each reference estimated value and each initial estimated value, a knowledge loss function is constructed;
[0044] Based on the knowledge loss function, parameters of the initial classification model are adjusted.
[0045] Optionally, the distillation unit is specifically used for:
[0046] Based on a preset mapping relationship, respectively mapping the ratio of each reference estimated value to the reference temperature coefficient, and mapping the ratio of each initial estimated value to the initial temperature coefficient, to obtain a reference coefficient corresponding to each reference estimated value, and an initial coefficient corresponding to each initial estimated value;
[0047] Based on the reference coefficients and the initial coefficients, the knowledge loss function is constructed.
[0048] Optionally, the distillation unit is specifically used for:
[0049] Based on the ratio of each reference coefficient to the sum of each reference coefficient, obtaining each reference ratio;
[0050] Based on the ratio of each of the initial coefficients to the sum of the initial coefficients, obtaining each predicted ratio;
[0051] The knowledge loss function is constructed based on the difference between each of the reference ratios and the corresponding predicted ratios, wherein the reference ratios and the corresponding predicted ratios are obtained based on the same first sample text.
[0052] Optionally, the device further includes a prediction unit, configured to:
[0053] Inputting the text to be classified into the target classification model;
[0054] Based on the target classification model, feature extraction is performed on the text to be classified to obtain text features of the text to be classified;
[0055] The text category of the text to be classified is predicted based on the text feature to obtain the target text category of the text to be classified.
[0056] An embodiment of the present application provides a text classification device, comprising:
[0057] An acquisition unit, used for acquiring text to be classified;
[0058] The prediction unit is used to input the text to be classified into a trained target classification model to obtain the target text category of the text to be classified, wherein the target classification model is obtained in the following manner:
[0059] Based on the first sample text set and the reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training;
[0060] Based on the second sample text set and the trained parameter prediction model, the intermediate classification model is subjected to multiple rounds of iterative training to obtain a target classification model. The parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting parameters of the intermediate classification model.
[0061] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned classification model training methods or text classification methods.
[0062] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the above-mentioned classification model training methods or text classification methods.
[0063] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned classification model training methods or text classification methods.
[0064] The beneficial effects of this application are as follows:
[0065] The embodiment of the present application provides a classification model training and text classification method, device, and electronic device, which can train a target classification model for text classification. First, based on a first sample text set and a reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model. Since the reference classification model already has strong capabilities in text classification tasks, the soft labels output by the reference classification model are used to adjust the parameters of the initial classification model during the knowledge distillation training. The gap between the reference classification model and the initial classification model can be quickly narrowed in a small number of knowledge distillation trainings, so that the intermediate classification model obtained can have a strong text classification capability. Based on this method, only a small amount of training data needs to be pre-labeled, which reduces the time spent on labeling data and improves the model training efficiency. In addition, the number of parameters of the intermediate classification model is greatly reduced compared to the reference classification model. When performing text classification tasks, the time required for text classification can be shortened, thereby improving the text classification efficiency.
[0066] Furthermore, after obtaining the intermediate classification model, the intermediate classification model is subjected to multiple rounds of iterative training based on the second sample text set and the trained parameter prediction model to obtain the target classification model. In each iterative training, the parameters of the intermediate classification model are adjusted by outputting a predicted parameter set through the parameter prediction model. Similarly, only a small amount of training data is needed to quickly improve the classification performance of the intermediate classification model, shorten the model training time, and improve the classification accuracy of the obtained target classification model, thereby improving the text classification accuracy when performing text classification tasks.
[0067] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0069] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;
[0070] Figure 2 This is a flowchart of a training method for a classification model in an embodiment of the present application;
[0071] Figure 3 A logical schematic diagram of a knowledge distillation method in an embodiment of the present application;
[0072] Figure 4 A schematic diagram of a training method for a meta-learning model in an embodiment of the present application;
[0073] Figure 5 A schematic diagram of a training method for a parameter adjustment model in an embodiment of the present application;
[0074] Figure 6 A schematic diagram of a training method for an intermediate classification model in an embodiment of the present application;
[0075] Figure 7 A logical schematic diagram of a joint training method in an embodiment of the present application;
[0076] Figure 8 A schematic diagram of the application process of a target classification model in an embodiment of the present application;
[0077] Fig. 9 is a flowchart of an implementation method of a text classification method in an embodiment of the present application;
[0078] Fig.10 This is a schematic diagram of the overall process of a text classification method in an embodiment of the present application;
[0079] Fig.11 A logical schematic diagram of a text classification method in an embodiment of the present application;
[0080] Fig.12 A schematic diagram of the structure of a training device for a classification model in an embodiment of the present application;
[0081] Fig.13 A schematic diagram of the structure of a text classification device in an embodiment of the present application;
[0082] Fig.14 A schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;
[0083] Fig.15 The present invention is a schematic diagram of the hardware structure of another electronic device to which the embodiments of the present application are applied. DETAILED DESCRIPTION
[0084] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.
[0085] The following is an introduction to some concepts involved in the embodiments of the present application.
[0086] Knowledge distillation: refers to extracting knowledge from a complex model and then passing it to a simplified model to improve the performance of the simplified model. The number of parameters of the simplified model is less than that of the complex model. In the embodiment of the present application, the knowledge distillation technology is used to extract knowledge from the reference classification model and pass it to the initial classification model, so that the obtained intermediate classification model can have better classification capabilities.
[0087] Reference classification model: used to guide the initial classification model for distillation training. It is a large-scale and complex model that has been trained. It has the same application direction as the initial classification model, that is, the reference model itself is a large-scale text classification model. In the embodiment of the present application, the reference classification model can also be called a teacher model, and the initial / intermediate classification model can be called a student model.
[0088] Initial classification model: It learns text classification by imitating the reference classification model. It has a small scale and a simple structure. After training, it can be applied to text classification scenarios.
[0089] Soft label: The model estimates the value of the sample text belonging to each preset text category. The model predicts the text type of the sample text and obtains a prediction result, which is based on an estimated value. For example, a text classification model predicts the sentiment type of a sample text. It predicts that the probability of the sample text being "positive" is 89%, and the probability of being other types is less than 89%. The model outputs a prediction result, which is that the sentiment type of the sample text is positive, and 89% is the estimated value.
[0090] Hard label: the text type of the sample text. Taking the above sample text as an example, the text type of this sample text is positive, and its hard label is "positive" or the identifier corresponding to "positive" (for example, a vector can be preset as the identifier).
[0091] Meta-Learning: refers to the machine learning algorithm itself can learn how to better adapt to a variety of different tasks or environments, also known as learning how to learn or learning-to-learn. In the embodiment of the present application, the parameters of the intermediate classification model are predicted by the parameter prediction model, which is the application of meta-learning technology, which can quickly improve the performance of the intermediate classification model.
[0092] The embodiments of the present application relate to artificial intelligence (AI), natural language processing (NLP) and machine learning (ML) technology, and are designed based on computer vision technology and machine learning in artificial intelligence.
[0093] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.
[0094] Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology mainly includes computer vision technology, natural language processing technology, and machine learning / deep learning, autonomous driving, smart transportation and other major directions. With the research and progress of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, robots, smart medical care, etc. It is believed that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role. The training of the classification model and the text classification method in the embodiment of the present application can be applied to fields such as autonomous driving, medical data recognition, and security systems. Combining artificial intelligence with text classification tasks in these fields can improve the efficiency of text classification.
[0095] In addition, the reference classification model, target classification model and parameter prediction model in the embodiments of the present application are obtained by training with machine learning or deep learning technology. Machine Learning (ML) is a multi-field cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.
[0096] After obtaining the target classification model based on the above technical training, the target classification model can be applied to classify texts to improve the accuracy of text classification.
[0097] The following is a brief introduction to the design concept of the embodiment of the present application:
[0098] Natural Language Processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It mainly studies various theories and methods that can enable effective communication between humans and computers using natural language.
[0099] Text classification is an important task in natural language processing, involving news classification, question-answer matching, intent recognition, inference, etc. With the development of deep learning technology, how to achieve text classification by training neural network models has become a hot issue that technicians are constantly exploring.
[0100] In related technologies, in order to train a text classification model to achieve the target task, a pre-trained model trained on a large-scale corpus is first obtained, and then a specific network layer of the pre-trained model is fine-tuned through the constructed sample text set to meet the requirements of the target task. However, the above method requires obtaining a large amount of training data for a specific task, and the training data for text classification tasks basically needs to be pre-labeled, which is time-consuming and has low model training efficiency. In addition, fine-tuning a specific network layer can easily cause the model to overfit, reducing the classification accuracy of the text classification model.
[0101] In view of this, the embodiments of the present application provide a classification model training and text classification method, device, and electronic device, which can train a target classification model for text classification. First, based on a first sample text set and a reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model. Since the reference classification model already has strong capabilities in text classification tasks, the soft labels output by the reference classification model are used to adjust the parameters of the initial classification model during the knowledge distillation training. The gap between the reference classification model and the initial classification model can be quickly narrowed in a small number of knowledge distillation trainings, so that the intermediate classification model obtained can have a strong text classification capability. Based on this method, only a small amount of training data needs to be pre-labeled, which reduces the time spent on labeling data and improves the model training efficiency. In addition, the number of parameters of the intermediate classification model is greatly reduced compared to the reference classification model. When performing text classification tasks, the time required for text classification can be shortened, thereby improving the text classification efficiency.
[0102] Furthermore, after obtaining the intermediate classification model, the intermediate classification model is subjected to multiple rounds of iterative training based on the second sample text set and the trained parameter prediction model to obtain the target classification model. In each iterative training, the parameters of the intermediate classification model are adjusted by outputting a predicted parameter set through the parameter prediction model. Similarly, only a small amount of training data is needed to quickly improve the classification performance of the intermediate classification model, shorten the model training time, and improve the classification accuracy of the obtained target classification model, thereby improving the text classification accuracy when performing text classification tasks.
[0103] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.
[0104] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.
[0105] In the embodiments of the present application, the terminal device includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; a text classification-related client may be installed on the terminal device, which may be software (such as a browser, model training software, etc.), or a web page, a small program, etc. The server is a background server corresponding to the software or web page, small program, etc., or a server specifically used for text classification, which is not specifically limited in the present application. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
[0106] It should be noted that the training method of the classification model in the embodiment of the present application can be executed by an electronic device, which can be a server or a terminal device, that is, the method can be executed by the server or the terminal device alone, or by the server and the terminal device together. For example, when executed by a server, the server performs multiple rounds of knowledge distillation training on the initial classification model based on the first sample text set and the reference classification model to obtain a trained intermediate classification model, and then, based on the second sample text set and the trained parameter prediction model, performs multiple rounds of iterative training on the intermediate classification model to obtain a target classification model. The text classification method in the embodiment of the present application can be applied to various text classification scenarios, such as article search engines, reading websites, etc., and the text to be classified is input into the target classification model to obtain the target text category of the text to be classified.
[0107] In an optional implementation, the terminal device and the server may communicate via a communication network.
[0108] In an optional implementation, the communication network is a wired network or a wireless network.
[0109] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.
[0110] In an embodiment of the present application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as in the training method of the classification model disclosed in the embodiment of the present application, the training data involved can be saved on the blockchain, for example, sample text, model parameters, model output, etc.
[0111] In addition, the embodiments of the present application can be applied to various scenarios, including not only text classification scenarios, but also including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.
[0112] The following describes the classification model training method and text classification method provided by the exemplary embodiment of the present application in combination with the application scenario described above and with reference to the accompanying drawings. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.
[0113] like Figure 2 As shown, it is an implementation flow chart of a classification model training method provided in an embodiment of the present application. Taking the execution subject as a server as an example, the specific implementation process of the method includes the following steps S21-S22:
[0114] S21: The server performs multiple rounds of knowledge distillation training on the initial classification model based on the first sample text set and the reference classification model to obtain a trained intermediate classification model;
[0115] Among them, the reference classification model is used to guide the initial classification model to perform knowledge distillation training. It is a large-scale and complex model that has been trained. It has the same application direction as the initial classification model, that is, the reference classification model itself is a large model for classifying text, for example, Bidirectional Encoder Representation from Transformers (BERT). Because the BERT model has shown extremely high performance and generalization ability in the field of NLP, the BERT model pre-trained on a large-scale corpus is used as the reference classification model and frozen at the level before the downstream task for knowledge distillation. In addition, the reference classification model can also be other pre-trained models that can be used for text classification, which are not specifically limited here.
[0116] The initial classification model can learn text classification by imitating the reference classification model. The obtained intermediate classification model is smaller in scale and simpler in structure than the reference classification model. The reference classification model can also be called a teacher model, and the initial classification model can be called a student model.
[0117] In addition, the training method of the model proposed in this application can be applied not only to text classification models, but also to other resource processing models, such as text classification models, text recognition models, text translation models, video classification models, image recognition models, etc. Regardless of the resource processing model, the application direction of the reference classification model is always the same as that of the resource processing model.
[0118] The server may use a randomly initialized text classification model as the initial classification model. The server may also pre-train the randomly initialized text classification model before performing distillation training, and use the pre-trained text classification model as the initial classification model.
[0119] For example, a simplified multi-layer perceptron (MLP) is selected as the student model. The MLP contains one or more fully connected layers and an output layer for predicting the label of the text. The MLP is initialized with random weights and used as the initial classification model, or the MLP is trained using the training set to obtain the initial weights of the MLP model, and the trained MLP is used as the initial classification model.
[0120] It should be noted that the network structure of the initial classification model used in the embodiments of the present application includes but is not limited to MLP, Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Graph Convolutional Neural Network (GCN) support vector machine, Bayesian model, etc., and is not specifically limited here.
[0121] The following is a brief introduction to MLP. The formula of MLP can be expressed by the following mathematical expression:
[0122] Taking an MLP with one fully connected layer and one output layer as an example, for an input x, after the transformation of the fully connected layer, the output h of the hidden layer is obtained:
[0123] h=σ(W1x+b1)
[0124] Among them, W1 is the weight of the fully connected layer, b1 is the bias term, and σ is the activation function. Commonly used activation functions include linear rectification function (ReLU), sigmoid function, hyperbolic tangent function (tanh), etc.
[0125] Next, the hidden layer output h is used as input, and after transformation by the output layer, the output y is obtained:
[0126] y=σ(W2h+b2)
[0127] Among them, W2 is the weight of the output layer, b2 is the bias term, and σ is the activation function. Normalized exponential function (softmax) is usually used as the activation function in classification tasks to convert the output into a probability distribution.
[0128] During the training process, the loss function of the model needs to be minimized. Common loss functions include cross entropy loss, mean square error, etc. For classification tasks, cross entropy loss is usually used, which can be expressed by the following formula:
[0129]
[0130] Where N is the number of training samples, C is the number of categories, and y ij represents the probability that the true label of the i-th sample is j, It represents the predicted probability of the model for the label j of the i-th sample. In the optimization process, the back propagation algorithm is used to calculate the gradient, and the optimizer is used to update the parameters of the model. Commonly used optimizers include stochastic gradient descent, Adam, etc.
[0131] In the prediction process, the input data is fed into the MLP to obtain the model's prediction results for the sample category. Usually, the category with the highest probability in the prediction results is taken as the final prediction result of the model.
[0132] In a round of knowledge distillation training, the reference classification model outputs soft labels for adjusting the parameters of the initial classification model. The soft labels include the reference estimated values of the first sample text corresponding to the reference classification model belonging to each preset text category. For example, it is expected to train a model that can perform multi-classification of the novel category to which the text belongs. The preset text categories are science fiction, suspense novels, and martial arts novels. The first sample text is "The intricate mystery will one day be solved by the world", and the soft labels output by the reference classification model are: science fiction 40%, suspense novels 80%, and martial arts novels 10%.
[0133] Optionally, obtain the soft labels output by the reference classification model by:
[0134] The reference classification model is used to extract features of the first sample text to obtain a high-dimensional feature vector. The text category of the first sample text is predicted based on the obtained feature vector to obtain a soft label.
[0135] Using soft labels to train the initial classification model can bring several benefits.
[0136] (1) Soft labels are richer than hard labels and can provide more training information to better guide the training of small models (initial classification models);
[0137] (2) Soft labels can reduce label noise and inconsistency because different categories may have fuzzy boundaries and intersections;
[0138] (3) Soft labels can make the model better adapt to different data distributions and task requirements, thereby improving the generalization ability and robustness of the model.
[0139] Knowledge distillation can be used as a preprocessing step to transfer the knowledge in the reference classification model to the initial classification model to improve the performance of the initial classification model and obtain an intermediate classification model. Then, we can use the meta-learning algorithm to further train and adjust the intermediate classification model to quickly adapt to the new task dataset and obtain an adaptively adjusted model.
[0140] S22: The server performs multiple rounds of iterative training on the intermediate classification model based on the second sample text set and the trained parameter prediction model to obtain a target classification model;
[0141] Among them, the sample texts contained in the second sample text set are not completely the same as those contained in the first sample text set. That is, in order to ensure the model training effect, the sample texts used in the knowledge distillation training and the iterative training are as different as possible. They can be partially the same or completely different.
[0142] Usually in the model training process, after obtaining the loss function of the model, the model parameters are adjusted according to the technician's experience or randomly, and then the direction of adjusting the model parameters is determined according to the changing trend of the loss function. This generally requires multiple rounds of iterations to effectively improve the model performance. In the present application, the parameter adjustment model is pre-trained and can accurately predict the parameters of the intermediate classification model. In one round of iterative training, the predicted parameter set for adjusting the parameters of the intermediate classification model is output. Therefore, only one or several iterative trainings are needed to quickly improve the performance of the intermediate classification model, shorten the model training time, and do not require a large amount of training data.
[0143] The prediction parameter set includes the parameters of the intermediate classification model predicted by the parameter adjustment model. For example, the parameters included in the prediction parameter set may be [a, b, c, d, e, f].
[0144] In an embodiment of the present application, first, based on the first sample text set and the reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model. Since the reference classification model already has strong capabilities in text classification tasks, the soft labels output by the reference classification model are used to adjust the parameters of the initial classification model during the knowledge distillation training. This can quickly narrow the gap between the reference classification model and the initial classification model in a small number of knowledge distillation trainings, so that the intermediate classification model obtained can have a strong text classification capability. Based on this method, only a small amount of training data needs to be pre-labeled, which reduces the time spent on labeling data and improves model training efficiency. In addition, the number of parameters of the intermediate classification model is greatly reduced compared to the reference classification model. When performing text classification tasks, the time required for text classification can be shortened, thereby improving text classification efficiency.
[0145] Furthermore, after obtaining the intermediate classification model, the intermediate classification model is subjected to multiple rounds of iterative training based on the second sample text set and the trained parameter prediction model to obtain the target classification model. In each iterative training, the parameters of the intermediate classification model are adjusted by outputting a predicted parameter set through the parameter prediction model. Similarly, only a small amount of training data is needed to quickly improve the classification performance of the intermediate classification model, shorten the model training time, and improve the classification accuracy of the obtained target classification model, thereby improving the text classification accuracy when performing text classification tasks.
[0146] Optionally, the following steps S31-S33 are performed in one round of knowledge distillation training:
[0147] S31: inputting the selected first sample text into the reference classification model and the initial classification model respectively, to obtain the soft label output by the reference classification model and the soft label output by the initial classification model;
[0148] S32: constructing a knowledge loss function based on each reference estimated value and each initial estimated value;
[0149] S33: Based on the knowledge loss function, adjust the parameters of the initial classification model.
[0150] The first sample text is input into the reference classification model and the initial classification model respectively. The reference classification model and the initial classification model perform the same task to predict the text category of the first sample text. Different tasks correspond to different preset text categories. For example, the preset text categories corresponding to the sentiment classification task include: positive and negative, the preset text categories corresponding to the novel classification task include: science fiction novels, suspense novels, martial arts novels, and the preset text categories for the news classification task include: finance, sports, and entertainment. Correspondingly, the soft labels output by the initial classification model include the initial estimated values of the first sample text belonging to each preset text category. For example, the soft labels output by the initial classification model are: 50% for science fiction novels, 60% for suspense novels, and 50% for martial arts novels.
[0151] In knowledge distillation training, it is necessary to minimize the distance between the teacher model and the student model. Therefore, a loss function can be constructed based on the difference between each reference estimate and each initial estimate, and parameters can be adjusted so that the estimate output by the initial classification model can gradually approach the estimate output by the reference classification model.
[0152] In this application, a knowledge distillation method based on a pre-trained model is proposed to transfer the knowledge in a complex model to a simplified model. A pre-trained BERT model can be used as a teacher model, and then its knowledge is distilled into a simple MLP model to improve the performance of the MLP model in downstream tasks. The knowledge distillation method can be used to transfer the knowledge of a large model to a small model, thereby improving the interpretability of the small model. In practical applications, there are often situations where the model needs to be explained or debugged. At this time, the small model has better interpretability and can help better understand the decision-making process and behavior of the model.
[0153] When constructing the knowledge loss function, KL divergence (Kullback–Leibler divergence) can be used as the objective function of knowledge distillation, and knowledge distillation can be achieved by minimizing the KL divergence loss function.
[0154]
[0155] Among them, P(y i ∣x i ) is the teacher model in the input x i The predicted category y i The probability distribution of Q(y i ∣x i ) is the student model in the input x i The predicted category y i The probability distribution of , N is the number of sample texts.
[0156] like Figure 3 As shown, it is a logical schematic diagram of a knowledge distillation method in an embodiment of the present application. The first sample text is "Today's sunset is so beautiful". The first sample text is input into the reference classification model and the initial classification model respectively. The soft labels output by the reference classification model are: positive 80%, negative 20%, and the soft labels output by the initial classification model are: positive 60%, negative 40%. Based on the difference between "positive 80%" and "positive 60%", and the difference between "negative 20%" and "negative 40%", a knowledge loss function is constructed, and then the parameters of the initial classification model are adjusted according to the knowledge loss function.
[0157] After obtaining the reference estimated value and the initial estimated value, different temperature parameters T can be used to smooth the probability distribution so that the student model can better learn the knowledge of the teacher model. Optionally, step S33 can be implemented as follows:
[0158] Based on the preset mapping relationship, the ratio of each reference estimated value to the reference temperature coefficient is mapped, and the ratio of each initial estimated value to the initial temperature coefficient is mapped to obtain the reference coefficient corresponding to each reference estimated value and the initial coefficient corresponding to each initial estimated value; based on each reference coefficient and each initial coefficient, a knowledge loss function is constructed.
[0159] Specifically, for a reference estimated value, first, the ratio of the reference estimated value to the reference temperature coefficient is calculated, the probability distribution of different reference estimated values is smoothed, and then the comparison value is mapped based on the preset mapping relationship to obtain the reference coefficient. The preset mapping relationship can be an exponential function, a logarithmic function, etc. Taking the preset mapping relationship as an exponential function as an example, the reference estimated value is 0.8, the reference temperature coefficient is 10, and the corresponding reference coefficient is e 0.08 , taking the preset mapping relationship as a logarithmic function as an example, the corresponding reference coefficient is lg0.08. Accordingly, the initial coefficient can be calculated based on the same method. It should be noted that the reference temperature coefficient and the initial temperature coefficient can be the same or different, and the specific value can be set according to actual needs.
[0160] After obtaining the reference coefficients and the initial coefficients, the loss function is calculated as follows:
[0161] Based on the ratio of each reference coefficient to the sum of each reference coefficient, each reference ratio is obtained; based on the ratio of each initial coefficient to the sum of each initial coefficient, each predicted ratio is obtained; based on the difference between each reference ratio and the corresponding predicted ratio, a knowledge loss function is constructed.
[0162] For example, if reference coefficient 1 is 0.8, reference coefficient 2 is 0.6, and reference coefficient 3 is 0.6, then reference ratio 1 is 0.8 / (0.8+0.6+0.6)=0.4, reference ratio 2 is 0.3, reference ratio 3 is 0.3, initial coefficient 1 is 0.5, initial coefficient 2 is 0.3, and initial coefficient 3 is 0.2, then prediction ratio 1 is 0.5, prediction ratio 2 is 0.3, and prediction ratio 3 is 0.2. In the knowledge distillation process, not only is it hoped that the text category finally output by the initial classification model is the same as the text category output by the reference classification model, but it is also hoped that the probability distribution of each preset text category predicted by the initial classification model is similar to the probability distribution of the reference classification model. Therefore, a knowledge loss function is constructed based on the difference between the reference ratio and the prediction ratio corresponding to the same first sample text, and the reference ratio and the corresponding prediction ratio are obtained based on the same first sample text.
[0163] Specifically, the reference ratio and predicted ratio can be calculated by the following formula:
[0164]
[0165]
[0166] Among them, P(y i ∣x i ) represents the reference ratio, Q(y i ∣x i ) represents the predicted ratio, f i and g i The teacher model and the student model are respectively i In the case of preset text category y i The logits of , C is the number of categories.
[0167] Based on the above method, since the reference classification model has been trained on a large-scale data set and has better generalization ability, these common features can be applied to specific tasks through knowledge distillation, thereby improving the generalization ability of the intermediate classification model. Since the reference classification model already contains rich feature representations, the time and resource consumption of model training can be reduced. Through the distillation method, the knowledge of the large model can be transferred to the small model, thereby accelerating the model training.
[0168] After the intermediate classification model is obtained in the above manner, the intermediate classification model can be further trained through the parameter prediction model. The server can directly obtain the trained parameter prediction model, or obtain the parameter prediction model by performing multiple rounds of training on the model to be trained before step S22.
[0169] Optionally, the parameter prediction model is trained through the following steps S41-S43:
[0170] S41: Obtain a model to be trained, multiple candidate classification models, and a third sample text set of each of the multiple candidate classification models;
[0171] S42: Based on each training sample text set and the model to be trained, multiple rounds of iterative training are performed on multiple candidate classification models to obtain each control classification model;
[0172] S43: Based on each control classification model and each verification sample text set, the training model is subjected to multiple rounds of iterative training to obtain a parameter prediction model.
[0173] Specifically, the model to be trained and the candidate classification model include but are not limited to MLP, LSTM, CNN, RNN, GCN, support vector machine, Bayesian model, etc., which are not specifically limited here. The network structure of each candidate classification model is not exactly the same. In this application, the model to be trained is mainly LSTM as an example for explanation. The network structure of the candidate classification model and the initial classification model may be different, but they can perform the same text classification task, for example, both are sentiment category classification of text.
[0174] Each candidate classification model corresponds to a third sample text set, which includes a training sample text set and a verification sample text set. The training sample text set is used to train the candidate classification model. During the iterative training process, the model to be trained predicts the parameters of the candidate classification model, and the performance of the candidate classification model is improved by updating the parameters. After the iterative training is completed, the control classification model obtained can be regarded as the prediction result of the model to be trained. According to the verification sample text data set, the model to be trained is trained to finally obtain a parameter prediction model.
[0175] like Figure 4As shown, it is a schematic diagram of a training method of a meta-learning model in an embodiment of the present application, wherein the meta-learning model refers to a parameter prediction model, the training sample text set 1 is input into the candidate classification model 1, and the output result 1 is obtained, the training sample text set 2 is input into the candidate classification model 2, and the output result 2 is obtained, the training sample text set 3 is input into the candidate classification model 3, and the output result 3 is obtained, the training model adjusts the parameters of the candidate classification model 1 based on the training sample text set 1 and the output result 1 to obtain the control classification model 1, and adjusts the parameters of the candidate classification model 2 based on the training sample text set 2 and the output result 2 to obtain the control classification model Type 2, based on the training sample text set 3 and the output result 3, the parameters of the candidate classification model 3 are adjusted to obtain the control classification model 3, then the verification sample text set 1 is input into the control classification model 1 to obtain the output result 1.1, the verification sample text set 2 is input into the control classification model 2 to obtain the output result 2.1, the verification sample text set 3 is input into the control classification model 3 to obtain the output result 3.1, based on the verification sample text set 1 and the output result 1.1, the verification sample text set 2 and the output result 2.1, the verification sample text set 3 and the output result 3.1, the parameters of the model to be trained are adjusted to obtain the parameter adjustment model.
[0176] Through meta-learning methods, we can quickly adapt to new tasks on small data sets and use existing knowledge to improve the accuracy of the model. In practical applications, we often encounter situations where data is insufficient or unevenly distributed. At this time, meta-learning methods can help quickly adapt to new data distributions and improve the accuracy of the model.
[0177] Optionally, in step S42, for multiple candidate classification models, in one round of iterative training, the following operations are performed respectively:
[0178] The selected training sample text is input into a candidate classification model to obtain the corresponding predicted text category; based on the difference between the predicted text category and the corresponding text category label, a classification loss function is constructed; based on the classification loss function and the current model parameters of a candidate classification model, the classification gradient is obtained; the classification gradient is input into the model to be trained for parameter prediction, an output prediction parameter set is obtained, and based on the prediction parameter set, parameters of a candidate classification model are adjusted.
[0179] Among them, the training sample text is set with a corresponding text category label. During the model training process, based on the difference between the predicted text category and the corresponding text category label, a classification loss function is constructed and the classification gradient is calculated. The trained model performs parameter prediction based on the classification gradient and outputs the parameters of the candidate classification model in the next iterative training.
[0180] Optionally, in a round of iterative training in step S43, the following operations are performed:
[0181] The verification sample texts selected from each verification sample text set are input into the corresponding control classification model to obtain the output prediction text categories; based on the difference between each prediction text category and the corresponding text category label, the verification loss function of each control classification model is constructed respectively; based on each verification loss function, the parameters of the model to be trained are adjusted.
[0182] Among them, the verification sample text is used to test the classification effect of the control classification model, which is actually to verify the parameter adjustment ability of the model to be trained. The verification sample text is set with a corresponding text category label. If the difference between the predicted text category output by the control classification model and the corresponding text category label is small, it indicates that the control classification model can achieve a better classification effect. It also reflects that the parameter adjustment ability of the model to be trained is strong. Through the constructed verification loss function, the parameters of the model to be trained are adjusted with the goal of minimizing the difference between the predicted text category and the corresponding text category label.
[0183] Specifically, in order to achieve meta-learning, an LSTM-based meta-learner can be used to learn how to adjust the knowledge transfer strategy between the reference classification model and the initial classification model. The learning rate is used as a control variable in the meta-learner to minimize the loss function on different tasks:
[0184]
[0185] Among them, L meta is the sum of the validation loss functions, represents the validation loss function on the j-th control classification model, f(θ j ) represents the j-th control classification model, and θ represents the model parameters.
[0186] like Figure 5 As shown, it is a schematic diagram of a training method for a parameter adjustment model in an embodiment of the present application. In one round of iterative training, the verification sample text 1 is input into the control classification model 1 to obtain the predicted text category "science fiction novel", and the corresponding text category label is "mystery novel". Based on the difference between "science fiction novel" and "mystery novel", the verification loss function 1 is constructed. The verification sample text 2 is input into the control classification model 2 to obtain the predicted text category "mystery novel", and the corresponding text category label is "mystery novel". Based on the difference between "mystery novel" and "mystery novel", the verification loss function 2 is constructed. The verification sample text 3 is input into the control classification model 3 to obtain the predicted text category "martial arts novel", and the corresponding text category label is "martial arts novel". Based on the difference between "martial arts novel" and "martial arts novel", the verification loss function 3 is constructed. With the goal of minimizing the sum of the verification loss function 1, the verification loss function 2 and the verification loss function 3, the parameters of the model to be trained are adjusted.
[0187] This application proposes a transfer learning method based on a meta-learning framework for fast and efficient learning on new tasks. An LSTM-based meta-learner is used to learn how to adjust the knowledge transfer strategy between the pre-trained model and the simplified model according to the characteristics of the new task. This meta-learning framework makes the model more adaptable in different tasks and environments and better able to cope with unknown data.
[0188] After the parameter adjustment model is obtained based on the above method, optionally, the following operations are performed in a round of iterative training in step S22:
[0189] The selected second sample text is input into the intermediate classification model to obtain the corresponding predicted text category; through the parameter prediction model, based on the target loss function constructed by the predicted text category and the corresponding text category label, parameter prediction is performed to obtain the output prediction parameter set; based on the prediction parameter set, the parameters of the intermediate classification model are adjusted.
[0190] Each second sample text is set with a corresponding text category label. For example, the text category labels of the second sample text include: finance, sports, entertainment, and technology. The predicted text categories output by the intermediate classification model include: finance, sports, entertainment, and technology. The target loss function can be a cross entropy loss, mean square error, etc., which is not specifically limited here. The parameter prediction model outputs a prediction parameter set in one round of iterative training. The parameters of the intermediate classification model are adjusted to the prediction parameter set for the next iterative training. Figure 6 As shown, it is a schematic diagram of a training method of an intermediate classification model in an embodiment of the present application. During a round of iterative training, the second sample text is "A technology company's stock price rose by more than 10% today", and the corresponding text category label is finance. The predicted text category output by the intermediate classification model is technology. A target loss function 1 is constructed based on finance and technology. The parameter prediction model performs parameter prediction based on the target loss function 1, and outputs a prediction parameter set [xyz np q] for adjusting the parameters of the intermediate classification model.
[0191] Optionally, based on the target loss function and the current model parameters of the intermediate classification model, a target gradient is obtained; the target gradient is input into a parameter prediction model for parameter prediction to obtain a prediction parameter set.
[0192] Specifically, in each iterative training, the gradient of the target loss function is calculated in combination with the current model parameters. The parameter prediction model combines these gradients to perform parameter prediction, outputs the predicted parameter set, and updates the current model parameters of the intermediate classification model.
[0193] The following is an example of a method for obtaining a target classification model using meta-learning technology after obtaining an intermediate classification model in an embodiment of the present application. Meta-learning is a method of learning that can quickly adapt to new tasks and improve the generalization ability of the model. In the embodiment of the present application, a model-agnostic meta-learning (MAML) algorithm is used to implement meta-learning:
[0194] (1) Input: MAML model (model to be trained), metadata dataset and task dataset. The metadata dataset contains multiple tasks, each of which contains a corresponding training set and validation set. The task dataset contains new tasks and corresponding training sets and test sets.
[0195] (2) Preprocess the metadata set. Divide the metadata set into several tasks, each of which contains a corresponding training set and a validation set. In practice, the original training set can be randomly sampled to divide the corresponding metadata sets.
[0196] (3) Train the MAML model. Train the MAML model on the metadata dataset to obtain the initial weights (parameters) of the MAML model;
[0197] (4) Train each task. For each task, use the MAML model to train and obtain a model for each task;
[0198] (5) Verify each task. For each task, use the validation set to verify the model of each task and obtain the performance index of each task (validation loss function);
[0199] (6) Update the MAML model. Update the weights of the MAML model based on the performance indicators of each task;
[0200] (7) Perform adaptive adjustments on the task dataset. For a new task dataset, use the MAML model to perform adaptive adjustments on a small number of samples to obtain the target classification model.
[0201] (8) Test on the test set. Use the adaptively adjusted model to perform sentiment analysis on the test set and obtain the final classification accuracy.
[0202] In the process of model training, there may be incompatibility issues between different data sources. Joint training can solve the incompatibility issues between different data sources and improve the accuracy of the target classification model. Joint training can integrate information from multiple data sources and learn a more comprehensive and accurate model from them. Joint training can reduce the risk of overfitting by sharing model parameters, thereby improving the generalization ability of the model.
[0203] Therefore, in the embodiment of the present application, joint training can also be integrated into the process of knowledge distillation and meta-learning, and the following strategies can be adopted:
[0204] (1) For knowledge distillation, joint training can be used as part of the distillation process to fuse the model parameters of different data sources. Multiple teacher models and student models can be trained on different data sources first, and then the multiple models can be fused using joint training. The model parameters of different data sources can be fused using a weighted average method, where the weights can be dynamically adjusted based on the size and quality of the data source. In this way, a more accurate and robust model can be obtained, thereby improving the performance of the model.
[0205] (2) For meta-learning, joint training can be used as part of the training process of the meta-learner (parameter adjustment model) to improve the generalization ability of the meta-learner. Multiple tasks can be trained on multiple data sources and combined using joint training to obtain a more accurate and robust meta-data set. This meta-data set is then used to train the meta-learner to better adapt to the data distribution and task requirements on the new task. In this way, a more adaptable meta-learner can be obtained, thereby improving the generalization ability of the model.
[0206] Specifically, multiple reference classification models are trained based on multiple first data sources (first sample text sets), and multiple initial classification models are subjected to multiple rounds of knowledge distillation training based on each reference classification model and the corresponding first sample text sets to obtain multiple trained intermediate classification models. Then, multiple reference prediction models are trained based on multiple third data sources, and multiple intermediate classification models are subjected to multiple rounds of iterative training based on multiple reference prediction models from the second sample text set to obtain multiple target classification models.
[0207] like Figure 7As shown, it is a logical schematic diagram of a joint training method in an embodiment of the present application. Data source 1, data source 2 and data source 3 are used to train reference classification model 1, reference classification model 2 and reference classification model 3 respectively. Further, reference classification model 1, reference classification model 2 and reference classification model 3 are used for knowledge distillation training to obtain intermediate classification model 1, intermediate classification model 2 and intermediate classification model 3. Finally, parameter adjustment model 1, parameter adjustment model 2 and parameter adjustment model 3 are used to adjust the parameters of the intermediate classification models to obtain target classification model 1, target classification model 2 and target classification model 3. In fact, the three models obtained by training based on the three data sources can be regarded as a model system in essence. For example, the final output result is determined by the output results of each model in the system. For example, text 1 is input into target classification model 1, target classification model 2 and target classification model 3 respectively. The category predicted by target classification model 1 is positive, the category predicted by target classification model 2 is negative, and the category predicted by target classification model 3 is positive. Then the output predicted text category is positive.
[0208] The following is a brief introduction to the implementation process of joint training. During the joint training process, stochastic gradient descent (SGD) is used to optimize the comprehensive loss function of the entire system to update the parameters of the reference classification model, the initial classification model, and the meta-learner. Small batch samples are used to train the model to speed up training and reduce memory consumption.
[0209] In the implementation of joint training, different hyperparameters are used to balance the loss functions of knowledge distillation, downstream tasks, and meta-learning. The following are the specific technical details:
[0210] Suppose there are k data sources, each with N k samples, each with n features and a target value. Suppose we want to jointly learn a multi-task regression model, where each task is to predict the target value. The prediction problem for each task can be expressed as:
[0211]
[0212] Among them, x i represents the n features of the i-th sample, represents the predicted target value of the i-th sample on the k-th data source, f k represents the prediction function of the kth data source, θ k Represents the model parameters of the k-th data source.
[0213] For the kth data source, its loss function can be defined as:
[0214]
[0215] in, Represents the predicted value and the true value y i The loss function between them, such as mean square error (MSE) or cross entropy (CE).
[0216] The goal is to minimize the sum of the loss functions of all data sources:
[0217]
[0218] Among them, α k Represents the weight of the kth data source, which is used to control the contribution of each data source to the total loss function. In multi-task learning, a weighted loss function is usually used to balance the contribution between different tasks.
[0219] Stochastic gradient descent (SGD) or other optimization algorithms can be used to minimize the total loss function.
[0220] For each data source, its gradient can be calculated, and then the optimization algorithm can be used to update its model parameters. For the kth data source, its gradient can be calculated using the back propagation algorithm:
[0221]
[0222] Then, an optimization algorithm is used to update the model parameters of the kth data source:
[0223]
[0224] Among them, η represents the learning rate, which is used to control the update step size of the parameters.
[0225] During the prediction process, the input data is fed into the models of all data sources to obtain the prediction results of each data source. Then, these prediction results can be fused using weighted average or other methods to obtain the final prediction result. The prediction results of each data source can be weighted averaged according to their weights:
[0226]
[0227] in, represents the final prediction result of the i-th sample, β k Represents the weight of the kth data source, which is used to control its contribution to the final prediction result. In multi-task learning, weighted averaging is usually used to balance the contribution between different tasks.
[0228] Finally, select appropriate hyperparameters to control the training and prediction of the model. In joint learning, you can select appropriate hyperparameters such as learning rate, momentum, weight decay, etc. to improve the training effect and generalization ability of the model. In addition, you can also use some techniques to alleviate overfitting and improve the robustness of the model, such as regularization, data enhancement, model fusion, etc.
[0229] After obtaining the target classification model, the text classification task is performed based on the target classification model, and the text to be classified is input into the target classification model; based on the target classification model, the feature extraction of the text to be classified is performed to obtain the text features of the text to be classified; based on the text features, the text category of the text to be classified is predicted to obtain the target text category of the text to be classified.
[0230] Specifically, the target classification model first extracts features from the text to be classified to obtain text features, then predicts the text category based on the text features and outputs the target text category. Figure 8 As shown, the text to be classified is "The semi-finals of this season will be held at the Fifth Gymnasium on the evening of April 28th", and the target classification model extracts features from the text to be classified, obtains text feature 1, and then performs text category prediction on text feature 1, outputting the target text category as sports.
[0231] like Fig. 9 As shown, it is an implementation flow chart of a text classification method provided in an embodiment of the present application. Taking the execution subject as a server as an example, the specific implementation process of the method includes the following steps S91-S92:
[0232] S91: Obtain the text to be classified;
[0233] S92: Input the text to be classified into the trained target classification model to obtain the target text category of the text to be classified.
[0234] Specifically, the target classification model is used to predict the text category of the text to be classified. The target classification model can perform various text classification tasks, such as news classification, article classification, sentiment classification, etc. The target text categories output are different for different text classification tasks. For example, when the target classification model performs a news classification task, the target text category output is finance, sports, or entertainment; when performing an article classification task, the target text category output is novel or poetry; when performing a sentiment classification task, the target text category output is positive or negative, which are not listed here one by one.
[0235] Among them, the target classification model is obtained by:
[0236] Based on the first sample text set and the reference classification model, the initial classification model is subjected to multiple rounds of knowledge distillation training to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training; based on the second sample text set and the trained parameter prediction model, the intermediate classification model is subjected to multiple rounds of iterative training to obtain a target classification model, and the parameter prediction model is used to: output a prediction parameter set for adjusting parameters of the intermediate classification model in one round of iterative training.
[0237] The process of obtaining the target classification model is as described in the above embodiment and will not be described in detail here.
[0238] like Fig.10 FIG. 1 is a schematic diagram of the overall process of a text classification method in an embodiment of the present application, comprising the following steps:
[0239] S1001: Obtain a model to be trained and multiple candidate classification models;
[0240] S1002: Based on each training sample text set and the model to be trained, perform multiple rounds of iterative training on multiple candidate classification models to obtain each control classification model;
[0241] S1003: Based on each control classification model and each verification sample text set, the training model is trained for multiple rounds to obtain a parameter prediction model;
[0242] S1004: Using the first sample text set and the reference classification model, perform multiple rounds of knowledge distillation training on the initial classification model to obtain an intermediate classification model;
[0243] S1005: Based on the second sample text set and the parameter prediction model, perform multiple rounds of iterative training on the intermediate classification model to obtain a target classification model;
[0244] S1006: Input the text to be classified into the target classification model;
[0245] S1007: The target classification model extracts features of the text to be classified to obtain text features of the text to be classified;
[0246] S1008: Obtain a target text category of the text to be classified based on text feature text category prediction.
[0247] like Fig.11As shown, it is a logical schematic diagram of a text classification method in an embodiment of the present application, taking the text classification task as sentiment classification as an example, that is, a text to be classified is input, and the target classification model needs to determine whether the text category is positive or negative. The reference classification model obtained is BERT, the initial classification model is MLP, the model to be trained is LSTM, the candidate classification model 1 is CNN, and the candidate classification model 2 is RNN. First, in the training process of candidate classification model 1 and candidate classification model 2, the model to be trained adjusts the parameters of candidate classification model 1 for N rounds to obtain the control classification model 1, and the model to be trained adjusts the parameters of candidate classification model 2 for N rounds to obtain the control classification model 2. The verification sample text set 1 is input into the control classification model 1 to obtain the verification loss function 1, and the verification sample text set 2 is input into the control classification model 2 to obtain the verification loss function 2. According to the verification loss function 1 and the verification loss function 2, the parameters of the model to be trained are adjusted to obtain the parameter prediction model. The initial classification model is trained through knowledge distillation by the reference classification model to obtain the intermediate classification model. The second sample text set is input into the intermediate classification model. The parameter prediction model adjusts the parameters of the intermediate classification model according to the output of the intermediate classification model to obtain the target classification model. The text to be classified "We will not give up no matter what difficulties we encounter" is input into the target classification model, and the text category is obtained as "positive".
[0248] Based on the same inventive concept, the present application also provides a training device for a classification model. Fig.12 As shown, it is a structural diagram of a training device 1200 for a classification model, which may include:
[0249] The distillation unit 1201 is used to perform multiple rounds of knowledge distillation training on the initial classification model based on the first sample text set and the reference classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training, and the soft labels include: reference estimated values of the first sample text corresponding to the reference classification model belonging to each preset text category;
[0250] The training unit 1202 is used to perform multiple rounds of iterative training on the intermediate classification model based on the second sample text set and the trained parameter prediction model to obtain a target classification model, wherein the sample texts included in the second sample text set are not completely the same as those included in the first sample text set; the parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting the parameters of the intermediate classification model.
[0251] Optionally, the training unit 1202 is further configured to:
[0252] Obtaining a model to be trained, multiple candidate classification models, and a third sample text set of each of the multiple candidate classification models, wherein the network structures of the multiple candidate classification models are not completely the same, and the third sample text set includes a training sample text set and a verification sample text set;
[0253] Based on each training sample text set and the model to be trained, multiple candidate classification models are respectively trained for multiple rounds of iterations to obtain each control classification model;
[0254] Based on each control classification model and each verification sample text set, the training model is trained for multiple rounds to obtain a parameter prediction model.
[0255] Optionally, each training sample text is provided with a corresponding text category label; then the training unit 1202 is specifically used for:
[0256] For multiple candidate classification models, in one round of iterative training, the following operations are performed respectively:
[0257] Input the selected training sample text into a candidate classification model to obtain the corresponding predicted text category;
[0258] Construct a classification loss function based on the difference between the predicted text category and the corresponding text category label;
[0259] Based on the classification loss function and the current model parameters of a candidate classification model, a classification gradient is obtained;
[0260] The classification gradient is input into the model to be trained for parameter prediction, an output prediction parameter set is obtained, and based on the prediction parameter set, parameters of a candidate classification model are adjusted.
[0261] Optionally, each verification sample text is provided with a corresponding text category label; then the training unit 1202 is specifically used for:
[0262] In one round of training iteration, the following operations are performed:
[0263] Input each verification sample text selected from each verification sample text set into the corresponding comparison classification model to obtain each predicted text category of the output;
[0264] Based on the difference between each predicted text category and the corresponding text category label, a validation loss function for each comparison classification model is constructed respectively;
[0265] Based on each validation loss function, the parameters of the model to be trained are adjusted.
[0266] Optionally, each second sample text is provided with a corresponding text category label; the training unit 1202 is specifically used for:
[0267] Input the selected second sample text into the intermediate classification model to obtain the corresponding predicted text category;
[0268] Through the parameter prediction model, the target loss function constructed based on the predicted text category and the corresponding text category label is used to perform parameter prediction and obtain the output prediction parameter set;
[0269] Based on the prediction parameter set, the parameters of the intermediate classification model are adjusted.
[0270] Optionally, the training unit 1202 is specifically configured to:
[0271] Obtain the target gradient based on the target loss function and the current model parameters of the intermediate classification model;
[0272] The target gradient is input into the parameter prediction model for parameter prediction to obtain a prediction parameter set.
[0273] Optionally, the distillation unit 1201 is specifically used for:
[0274] In one round of knowledge distillation training, the following operations are performed:
[0275] Inputting the selected first sample text into the reference classification model and the initial classification model respectively, obtaining the soft label output by the reference classification model and the soft label output by the initial classification model, wherein the soft label output by the initial classification model includes: the initial estimated value of each preset text category that the first sample text belongs to;
[0276] Based on each reference estimated value and each initial estimated value, a knowledge loss function is constructed;
[0277] Based on the knowledge loss function, the parameters of the initial classification model are adjusted.
[0278] Optionally, the distillation unit 1201 is specifically used for:
[0279] Based on the preset mapping relationship, the ratio of each reference estimated value to the reference temperature coefficient is mapped, and the ratio of each initial estimated value to the initial temperature coefficient is mapped, so as to obtain the reference coefficient corresponding to each reference estimated value and the initial coefficient corresponding to each initial estimated value;
[0280] Based on each reference coefficient and each initial coefficient, a knowledge loss function is constructed.
[0281] Optionally, the distillation unit 1201 is specifically used for:
[0282] Based on the ratio of each reference coefficient to the sum of each reference coefficient, each reference ratio is obtained;
[0283] Based on the ratio of each initial coefficient to the sum of each initial coefficient, each predicted ratio is obtained;
[0284] A knowledge loss function is constructed based on the difference between each reference ratio and the corresponding predicted ratio, wherein the reference ratio and the corresponding predicted ratio are obtained based on the same first sample text.
[0285] Optionally, the device further includes a prediction unit 1203, configured to:
[0286] Input the text to be classified into the target classification model;
[0287] Based on the target classification model, feature extraction is performed on the text to be classified to obtain text features of the text to be classified;
[0288] The text category of the text to be classified is predicted based on the text features to obtain the target text category of the text to be classified.
[0289] Based on the same inventive concept, the present application also provides a text classification device. Fig.13 As shown, it is a structural diagram of a text classification device 1300, which may include:
[0290] An acquisition unit 1301 is used to acquire the text to be classified;
[0291] The prediction unit 1302 is used to input the text to be classified into a trained target classification model to obtain the target text category of the text to be classified, wherein the target classification model is obtained in the following manner:
[0292] Based on the first sample text set and the reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training;
[0293] Based on the second sample text set and the trained parameter prediction model, the intermediate classification model is iteratively trained for multiple rounds to obtain a target classification model. The parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting parameters of the intermediate classification model.
[0294] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.
[0295] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.
[0296] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.14 As shown, it includes a memory 1401 , a communication module 1403 and one or more processors 1402 .
[0297] The memory 1401 is used to store computer programs executed by the processor 1402. The memory 1401 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.
[0298] The memory 1401 may be a volatile memory, such as a random-access memory (RAM); the memory 1401 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1401 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1401 may be a combination of the above memories.
[0299] The processor 1402 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1402 is used to implement the above classification model training method or text classification method when calling the computer program stored in the memory 1401.
[0300] The communication module 1403 is used to communicate with terminal devices and other servers.
[0301] The specific connection medium between the memory 1401, the communication module 1403 and the processor 1402 is not limited in the embodiment of the present application. Fig.14 In the embodiment, the memory 1401 and the processor 1402 are connected via a bus 1404. Fig.14 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1404 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.14 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.
[0302] The memory 1401 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the classification model training method or text classification method of the embodiment of the present application. The processor 1402 is used to execute the above-mentioned classification model training method or text classification method, such as Figure 2 or Fig. 9 shown.
[0303] In another embodiment, the electronic device may also be other electronic devices, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.15 As shown, it includes: a communication component 1510, a memory 1520, a display unit 1530, a camera 1540, a sensor 1550, an audio circuit 1560, a Bluetooth module 1570, a processor 1580 and other components.
[0304] The communication component 1510 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the user to send and receive information through the WiFi module.
[0305] The memory 1520 can be used to store software programs and data. The processor 1580 executes various functions and data processing of the terminal device by running the software programs or data stored in the memory 1520. The memory 1520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1520 stores an operating system that enables the terminal device to run. In the present application, the memory 1520 can store an operating system and various application programs, and can also store a computer program for executing the training method of the classification model or the text classification method in the embodiment of the present application.
[0306] The display unit 1530 can also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device. Specifically, the display unit 1530 may include a display screen 1532 disposed on the front of the terminal device. The display screen 1532 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 1530 may be used to display the text classification user interface in the embodiment of the present application, etc.
[0307] The display unit 1530 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device. Specifically, the display unit 1530 may include a touch screen 1531 arranged on the front of the terminal device, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.
[0308] The touch screen 1531 can be covered on the display screen 1532, or the touch screen 1531 and the display screen 1532 can be integrated to realize the input and output functions of the terminal device, and the integrated display screen can be referred to as a touch display screen. In this application, the display unit 1530 can display the application program and the corresponding operation steps.
[0309] The camera 1540 can be used to capture still images, and the user can post comments on the images taken by the camera 1540 through the application. The camera 1540 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the processor 1580 to convert it into a digital image signal.
[0310] The terminal device may further include at least one sensor 1550, such as an acceleration sensor 1551, a distance sensor 1552, a fingerprint sensor 1553, and a temperature sensor 1554. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.
[0311] The audio circuit 1560, the speaker 1561, and the microphone 1562 can provide an audio interface between the user and the terminal device. The audio circuit 1560 can transmit the electrical signal converted from the received audio data to the speaker 1561, which is converted into a sound signal for output. The terminal device can also be equipped with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1560 and converted into audio data, and then the audio data is output to the communication component 1510 to be sent to, for example, another terminal device, or the audio data is output to the memory 1520 for further processing.
[0312] The Bluetooth module 1570 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1570 to exchange data.
[0313] The processor 1580 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1520 and calling data stored in the memory 1520. In some embodiments, the processor 1580 may include one or more processing units; the processor 1580 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, and the baseband processor mainly processes wireless communication. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1580. In the present application, the processor 1580 can run the operating system, the application program, the user interface display and the touch response, as well as the training method of the classification model or the text classification method of the embodiment of the present application. In addition, the processor 1580 is coupled to the display unit 1530.
[0314] In some possible implementations, various aspects of the classification model training method or text classification method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the classification model training method or text classification method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 2 or Fig. 9 Follow the steps shown in .
[0315] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0316] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.
[0317] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.
[0318] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0319] Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The computer program may be executed entirely on the user electronic device, partially on the user device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device may be connected to the user electronic device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).
[0320] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.
[0321] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0322] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.
[0323] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0324] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0325] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0326] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0327] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A classification model training method, characterized in that: The method comprises: Based on the first sample text set and the reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: in one round of knowledge distillation training, output soft labels for adjusting parameters of the initial classification model, and the soft labels include: reference estimated values of the first sample text corresponding to the reference classification model belonging to each preset text category; Based on the second sample text set and the trained parameter prediction model, the intermediate classification model is subjected to multiple rounds of iterative training to obtain a target classification model, wherein the sample texts contained in the second sample text set are not completely the same as those contained in the first sample text set; the parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting the parameters of the intermediate classification model.
2. The method according to claim 1, characterized in that The parameter prediction model is trained in the following way: Obtain a model to be trained, multiple candidate classification models, and a third sample text set of each of the multiple candidate classification models, wherein the network structures of the multiple candidate classification models are not completely the same, and the third sample text set includes a training sample text set and a verification sample text set; Based on each training sample text set and the model to be trained, the multiple candidate classification models are respectively subjected to multiple rounds of iterative training to obtain each control classification model; Based on the control classification models and the verification sample text sets, the model to be trained is subjected to multiple rounds of iterative training to obtain the parameter prediction model.
3. The method according to claim 2, characterized in that Each training sample text is set with a corresponding text category label; Then, based on each training sample text set and the model to be trained, the multiple candidate classification models are respectively subjected to multiple rounds of iterative training to obtain each control classification model, including: For the multiple candidate classification models, in one round of iterative training, the following operations are performed respectively: Input the selected training sample text into a candidate classification model to obtain the corresponding predicted text category; Constructing a classification loss function based on the difference between the predicted text category and the corresponding text category label; Obtaining a classification gradient based on the classification loss function and a current model parameter of the candidate classification model; The classification gradient is input into the model to be trained for parameter prediction to obtain an output prediction parameter set, and based on the prediction parameter set, parameters of the candidate classification model are adjusted.
4. The method according to claim 2, characterized in that Each verification sample text is set with a corresponding text category label; Then, based on the reference classification models and the verification sample text sets, the model to be trained is subjected to multiple rounds of iterative training to obtain the parameter prediction model, including: In one round of training iterations, the following operations are performed: Inputting each verification sample text selected from each verification sample text set into a corresponding comparison classification model to obtain output each predicted text category; Based on the difference between each predicted text category and the corresponding text category label, constructing a verification loss function for each comparison classification model; Based on the verification loss functions, parameters of the model to be trained are adjusted.
5. The method according to claim 1, characterized in that Each second sample text is set with a corresponding text category label; the following operations are performed in one round of iterative training: Inputting the selected second sample text into the intermediate classification model to obtain a corresponding predicted text category; By using the parameter prediction model, a target loss function constructed based on the predicted text category and the corresponding text category label is used to perform parameter prediction to obtain an output prediction parameter set; Based on the prediction parameter set, parameters of the intermediate classification model are adjusted.
6. The method according to claim 5, characterized in that The parameter prediction model is used to perform parameter prediction based on the target loss function constructed based on the predicted text category and the corresponding text category label to obtain an output prediction parameter set, including: Obtaining a target gradient based on the target loss function and current model parameters of the intermediate classification model; The target gradient is input into the parameter prediction model to perform parameter prediction to obtain the prediction parameter set.
7. The method according to any one of claims 1 to 6, characterized in that: In one round of knowledge distillation training, the following operations are performed: Inputting the selected first sample text into the reference classification model and the initial classification model respectively, obtaining a soft label output by the reference classification model and a soft label output by the initial classification model, wherein the soft label output by the initial classification model includes: an initial estimated value of each of the preset text categories to which the first sample text belongs; Based on each reference estimated value and each initial estimated value, a knowledge loss function is constructed; Based on the knowledge loss function, parameters of the initial classification model are adjusted.
8. The method according to claim 7, characterized in that The knowledge loss function is constructed based on each reference estimated value and each initial estimated value, including: Based on a preset mapping relationship, respectively mapping the ratio of each reference estimated value to the reference temperature coefficient, and mapping the ratio of each initial estimated value to the initial temperature coefficient, to obtain a reference coefficient corresponding to each reference estimated value, and an initial coefficient corresponding to each initial estimated value; Based on the reference coefficients and the initial coefficients, the knowledge loss function is constructed.
9. The method according to claim 8, characterized in that The constructing the knowledge loss function based on the reference coefficients and the initial coefficients includes: Based on the ratio of each reference coefficient to the sum of each reference coefficient, obtaining each reference ratio; Based on the ratio of each of the initial coefficients to the sum of the initial coefficients, obtaining each predicted ratio; The knowledge loss function is constructed based on the difference between each of the reference ratios and the corresponding predicted ratios, wherein the reference ratios and the corresponding predicted ratios are obtained based on the same first sample text.
10. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Inputting the text to be classified into the target classification model; Based on the target classification model, feature extraction is performed on the text to be classified to obtain text features of the text to be classified; The text category of the text to be classified is predicted based on the text feature to obtain the target text category of the text to be classified.
11. A text classification method, characterized in that: The method comprises: Get the text to be classified; The text to be classified is input into a trained target classification model to obtain a target text category of the text to be classified, wherein the target classification model is obtained in the following manner: Based on the first sample text set and the reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training; Based on the second sample text set and the trained parameter prediction model, the intermediate classification model is subjected to multiple rounds of iterative training to obtain a target classification model. The parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting parameters of the intermediate classification model.
12. A training device for a classification model, characterized in that: include: A distillation unit is used to perform multiple rounds of knowledge distillation training on the initial classification model based on the first sample text set and the reference classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training, and the soft labels include: reference estimated values of the first sample text corresponding to the reference classification model belonging to each preset text category; A training unit is used to perform multiple rounds of iterative training on the intermediate classification model based on a second sample text set and a trained parameter prediction model to obtain a target classification model, wherein the sample texts contained in the second sample text set are not completely the same as those contained in the first sample text set; and the parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting parameters of the intermediate classification model.
13. A text classification device, characterized in that: include: An acquisition unit, used for acquiring the text to be classified; The prediction unit is used to input the text to be classified into a trained target classification model to obtain the target text category of the text to be classified, wherein the target classification model is obtained in the following manner: Based on the first sample text set and the reference classification model, multiple rounds of knowledge distillation training are performed on the initial classification model to obtain a trained intermediate classification model, wherein the reference classification model is used to: output soft labels for adjusting parameters of the initial classification model in one round of knowledge distillation training; Based on the second sample text set and the trained parameter prediction model, the intermediate classification model is subjected to multiple rounds of iterative training to obtain a target classification model. The parameter prediction model is used to: in one round of iterative training, output a prediction parameter set for adjusting parameters of the intermediate classification model.
14. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 11.
15. A computer-readable storage medium, characterized in that: The method comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the methods of claims 1 to 11.
16. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the methods described in claims 1 to 11.
Citation Information
Cited By
Classification and question and answer large model joint parameter adjustment method and device
CN119621912A
Text generation method and device based on intelligent agent, equipment and storage medium
CN119740669A
Method and apparatus for managing machine learning model on basis of distillation, device and medium
WO2026156505A1
