Text classification method, text classification device, system and readable storage medium

Abstract

The application provides a text classification method, a text classification device, a system and a readable storage medium. The text classification method comprises the following steps: determining a first model and a plurality of second models from a plurality of preselected classification models according to a text classification parameter; training the plurality of second models according to first text data, and determining a plurality of output data of the plurality of second models after training; training the first model according to the plurality of output data and the first text data; performing optimization processing on the first model after training, and performing compression processing on the first model after optimization processing; performing text classification processing on a text to be classified according to the compressed first model; and the optimization processing is used for improving the training degree of the first model.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more specifically, to a text classification method, a text classification device, a text classification system, and a readable storage medium. Background Technology

[0002] In existing technologies, with the continuous advancement of natural language processing and knowledge distillation techniques, large-scale text classification models are widely used in text classification tasks. However, in scenarios such as search, recommendation, and dialogue, it is necessary to classify the text data appearing in the scenario to ensure the smooth progress of the task. Existing text classification models, when classifying text in these scenarios, mainly suffer from the following problems:

[0003] (1) When the text data to be classified has many text categories and a small number of texts in each category, the classification effect of the text classification model will be reduced due to insufficient feature information corresponding to each text category.

[0004] (2) Text classification models with good text classification performance have a lot of model parameters. In specific classification tasks, text classification models are prone to redundancy, which affects the classification speed of the model in specific classification tasks and increases the consumption of resources in the classification process. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0006] Therefore, one aspect of the present invention is to propose a text classification method.

[0007] Another aspect of the present invention is to provide a text classification device.

[0008] Another aspect of the present invention is to propose a text classification system.

[0009] Another aspect of the present invention is to provide a readable storage medium.

[0010] In view of this, according to one aspect of the present invention, a text classification method is proposed, comprising: determining a first model and a plurality of second models from a plurality of pre-selected classification models based on text classification parameters; training the plurality of second models based on first text data and determining a plurality of output data of the trained plurality of second models; training the first model based on the plurality of output data and the first text data; optimizing the trained first model and compressing the optimized first model; performing text classification processing on the text to be classified based on the compressed first model; the optimization processing is used to improve the training degree of the first model.

[0011] The text classification method provided by this invention first determines a first model and multiple second models from multiple pre-selected classification models based on text classification parameters. After determining the classification models, the multiple second models are trained using first text data to obtain multiple output data of the multiple second models corresponding to the first text data. The first model is then trained based on the multiple output data and the first text data. The trained first model is then optimized to enhance its training level. The optimized first model is then compressed to reduce the number of parameters. Finally, the compressed first model is used as the text classification model to classify the text data in the text classification task.

[0012] Specifically, the first model can be a student model, and the second model can be a teacher model. The text classification method provided by this invention selects the second model through text classification parameters, ensuring the classification effect of the selected second model and thus better training the first model. The first model, trained with multiple second models, learns from their classification experience without needing extensive retraining, improving its text classification ability. This allows it to maintain excellent classification performance even with a small number of texts and many text categories, and ensures consistent excellent classification performance across different evaluation metrics. Optimizing the trained first model improves its text classification, reducing the probability of classification errors and further enhancing classification effectiveness. Compressing the optimized first model reduces the number of parameters, avoiding redundancy in specific classification scenarios. This improves classification speed while reducing resource consumption during text classification tasks, thus enhancing the user experience.

[0013] The text classification method of the present invention may also have the following technical features:

[0014] In the above technical solution, before training multiple second models based on the first text data, the method further includes: configuring first parameters for the first model so that the first model can be trained based on the first parameters, multiple output data, and the first text data.

[0015] In this technical solution, the first parameter can be a hyperparameter. After selecting the first model and multiple second models, the first parameter is configured for the first model. By configuring the first parameter, the training performance and training effect of the first model are increased, thereby improving the text classification effect of the trained first model.

[0016] In any of the above technical solutions, determining a first model and multiple second models from multiple pre-selected classification models based on text classification parameters specifically includes: training each of the multiple pre-selected classification models based on the second text data; calculating the text classification parameters corresponding to each trained pre-selected classification model; and determining the first model and multiple second models from the multiple pre-selected classification models based on the parameter values ​​of the text classification parameters corresponding to each pre-selected classification model.

[0017] In this technical solution, each pre-selected classification model is trained using second text data, ensuring fairness in selection by training each model on a single text dataset. This also facilitates the statistical analysis of the text classification parameters for each pre-selected model. By using the text classification parameters corresponding to each pre-selected model, the classification performance of each model can be more intuitively obtained, increasing the accuracy of judging its classification effectiveness. Furthermore, by using the text classification parameter values ​​as the selection criteria for the first model and multiple second models, the classification performance of the first model and multiple second models is guaranteed, thereby enhancing the training effect of the second models on the first model.

[0018] In any of the above technical solutions, the text classification parameters include at least one of the following or a combination thereof: accuracy, precision, recall, and F1 score.

[0019] In this technical solution, text classification parameters include one or more of the following: accuracy, precision, recall, and F1 score. Accuracy represents the proportion of correctly predicted results out of the total number of classified samples; precision represents the proportion of actually positive samples among those predicted as positive; recall represents the proportion of actually positive samples among those predicted as positive; and the F1 score is the harmonic mean of precision and recall, meaning it considers both the model's precision and recall. These four parameters represent the specific classification performance of the model.

[0020] In any of the above technical solutions, the optimization processing of the trained first model specifically includes: configuring a first preset function for the first text data and configuring multiple second preset functions for multiple second output data; configuring a first weight for the first preset function and configuring a second weight for the multiple second preset functions; performing a weighted summation of the first preset function and the multiple second preset functions according to the first weight and the second weight to obtain a first processing function; and optimizing the first model according to the first processing function.

[0021] In this technical solution, different preset functions are configured for the first text data and multiple second output data respectively. Specifically, a first preset function is configured for the first text data and a second preset function is configured for the multiple second output data. Subsequently, different weights are configured for the different preset functions, and the preset functions with configured weights are weighted and summed to obtain the final first processing function. The first processing function is then used to optimize the first model.

[0022] Specifically, both the first and second preset functions are loss functions. By configuring different loss functions for the first text data and multiple second output data, different weights are assigned to different loss functions, and a weighted sum is performed based on the weights of the loss functions themselves. The weighted sum is then used as the loss function of the first model. This avoids the gradient explosion phenomenon caused by the continuous accumulation of error gradients during the training of the first model, ensuring the stability of the first model during operation, thereby improving the training effect and the model performance of the first model, and thus achieving the optimization of the first model.

[0023] In any of the above technical solutions, after determining the first model and multiple second models from multiple pre-selected classification models based on text classification parameters, the method further includes: configuring a different database for each of the multiple second models.

[0024] In this technical solution, after determining the first model and multiple second models, a different database is configured for each second model. Specifically, the database can be a lexicon. The lexicon is used by the word segmenter corresponding to the second model. When different lexicons are used, the vocabulary size and vocabulary range of the second model are also different. Therefore, selecting different vocabulary sizes for each second model allows the second model to have different vocabulary ranges, thereby enabling the first model to learn a wider range of vocabulary during training, increasing the training effect of the first model and the word segmentation effect of the first model after training, and thus improving the first model's ability to learn different words.

[0025] In any of the above technical solutions, the first model after optimization is compressed, specifically including: performing knowledge distillation on the first model after optimization; and performing adaptive processing on the first model after knowledge distillation.

[0026] In this technical solution, firstly, knowledge distillation is performed on the optimized first model, thereby reducing the number of parameters of the model itself without losing the information obtained by the first model through training. This improves the classification speed of the model in specific classification scenarios while reducing the resource consumption when performing text classification tasks. By adaptively processing the knowledge-distilled first model, the model structure can be adjusted according to the different requirements of classification accuracy and classification speed for different classification tasks, thereby improving the user experience.

[0027] According to another aspect of the present invention, a text classification apparatus is provided, comprising: a determining unit, configured to determine a first model and a plurality of second models from a plurality of pre-selected classification models based on text classification parameters; a processing unit, configured to train the plurality of second models based on first text data, and determine a plurality of output data of the trained plurality of second models; the processing unit is further configured to: train the first model based on the plurality of output data and the first text data; optimize the trained first model, and compress the optimized first model; perform text classification processing on the text to be classified based on the compressed first model; wherein the optimization processing is used to improve the training degree of the first model.

[0028] The text classification apparatus provided by this invention includes a determining unit and a processing unit. The determining unit is used to determine a first model and multiple second models from multiple pre-selected classification models based on text classification parameters. Once the determining unit has determined the first model and multiple second models, the processing unit first trains the multiple second models using first text data to obtain multiple output data corresponding to the first text data. Then, it trains the first model based on the multiple output data and the first text data, and optimizes the trained first model to enhance its training level. Subsequently, it compresses the optimized first model to reduce the number of parameters, and uses the compressed first model as the text classification model to classify the text data in the text classification task.

[0029] Specifically, the first model can be a student model, and the second model can be a teacher model. The text classification method provided by this invention selects the second model through text classification parameters, ensuring the classification effect of the selected second model and thus better training the first model. The first model, trained with multiple second models, learns from their classification experience without needing extensive retraining, improving its text classification ability. This allows it to maintain excellent classification performance even with a small number of texts and many text categories, and ensures consistent excellent classification performance across different evaluation metrics. Optimizing the trained first model improves its text classification, reducing the probability of classification errors and further enhancing classification effectiveness. Compressing the optimized first model reduces the number of parameters, avoiding redundancy in specific classification scenarios. This improves classification speed while reducing resource consumption during text classification tasks, thus enhancing the user experience.

[0030] According to another aspect of the present invention, a text classification system is proposed, comprising a memory storing a program; and a processor that executes the program to implement the text classification method as described in any of the above technical solutions.

[0031] The text classification system provided by this invention includes a memory and a processor. The memory stores a program; the processor is connected to the memory and executes the program, implementing the text classification method as described in any of the above technical solutions when executing the program. Therefore, it possesses all the beneficial effects of the text classification method as described in any of the above technical solutions.

[0032] According to another aspect of the present invention, a readable storage medium is provided on which a program is stored, which, when executed by a processor, implements the text classification method as described in any of the above-described technical solutions.

[0033] The readable storage medium provided by the present invention is used to store a program that implements the text classification method as described in any one of the above technical solutions when executed by a processor. Therefore, it has all the beneficial effects of the text classification method as described in any one of the above technical solutions.

[0034] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0035] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0036] Figure 1 One of the flowcharts of the text classification method according to an embodiment of the present invention is shown;

[0037] Figure 2 The second flowchart of the text classification method according to an embodiment of the present invention is shown;

[0038] Figure 3 The third flowchart of the text classification method according to an embodiment of the present invention is shown;

[0039] Figure 4 The fourth flowchart of the text classification method according to an embodiment of the present invention is shown;

[0040] Figure 5 The fifth flowchart of the text classification method according to an embodiment of the present invention is shown;

[0041] Figure 6 The sixth flowchart of the text classification method according to an embodiment of the present invention is shown;

[0042] Figure 7 This diagram illustrates a flowchart of the text classification method according to an embodiment of the present invention in a specific application scenario.

[0043] Figure 8 A structural block diagram of a text classification device according to an embodiment of the present invention is shown. Detailed Implementation

[0044] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0046] The following reference Figures 1 to 8 This invention describes a text classification method, a text classification apparatus, a text classification system, and a readable storage medium according to some embodiments of the present invention.

[0047] Example 1:

[0048] One embodiment of the present invention proposes a text classification method. Figure 1 One of the flowcharts of a text classification method according to an embodiment of this application is shown, such as... Figure 1 As shown, text classification methods include:

[0049] Step 102: Determine the first model and multiple second models from multiple pre-selected classification models based on the text classification parameters;

[0050] Step 104: Train multiple second models based on the first text data, and determine multiple second output data of the trained multiple second models;

[0051] Step 106: Train the first model based on multiple second output data and the first text data;

[0052] Step 108: Optimize the trained first model and compress the optimized first model.

[0053] Step 110: Perform text classification processing on the text to be classified based on the compressed first model.

[0054] In this embodiment, a first model and multiple second models are first determined from multiple pre-selected classification models based on text classification parameters. After the classification models are determined, the multiple second models are trained using the first text data to obtain multiple output data of the multiple second models corresponding to the first text data. The first model is then trained based on the multiple output data and the first text data. The trained first model is then optimized to enhance its training level. The optimized first model is then compressed to reduce the number of parameters. The compressed first model is then used as the text classification model to classify the text data in the text classification task.

[0055] Specifically, the first model can be a student model, and the second model can be a teacher model. The text classification method defined in this embodiment selects the second model based on text classification parameters, ensuring the classification performance of the selected second model and thus better training the first model. The first model, trained with multiple second models, learns from their classification experience without requiring extensive retraining, improving its text classification ability. This allows it to maintain excellent classification performance even with a small number of texts and many text categories, while also achieving consistently excellent classification performance across different evaluation metrics. Optimizing the trained first model improves its text classification, reducing the probability of classification errors and further enhancing its performance. Compression of the optimized first model reduces the number of parameters, avoiding redundancy in specific classification scenarios. This improves classification speed while reducing resource consumption during text classification tasks, thus enhancing the user experience.

[0056] For example, in the embodiments of this application, the selection of the pre-selected classification model should meet the requirement that the selected pre-selected classification model has text classification function. Based on the above conditions, the embodiments of this application use the bert-wwm-ext-chinese model (a Chinese pre-trained language model released by Harbin Institute of Technology and iFlytek Laboratory), the hfl / roberta-wwm-ext model (a Chinese pre-trained model released by Harbin Institute of Technology and iFlytek Laboratory in combination with the Roberta model), the xlnet-base-cased model (an unsupervised speech representation learning method model released by Google), the gpt2-medium-en model (a language model released by OpenAI), the ernie-1.0 model (an optimized model based on the bert model by Baidu), and the ernie-gram-zh model (a multi-granularity language knowledge enhancement model by Baidu) as pre-selected classification models.

[0057] Example 2

[0058] An embodiment of the present invention provides a text classification method, such as... Figure 2 As shown, text classification methods include:

[0059] Step 202: Determine the first model and multiple second models from multiple pre-selected classification models based on the text classification parameters;

[0060] Step 204: Configure the first parameters for the first model;

[0061] Step 206: Train multiple second models based on the first text data, and determine multiple second output data of the trained multiple second models;

[0062] Step 208: Train the first model based on multiple second output data and the first text data;

[0063] Step 210: Optimize the trained first model and compress the optimized first model.

[0064] Step 212: Perform text classification processing on the text to be classified based on the compressed first model.

[0065] In this embodiment, after selecting the first model and multiple second models, a first parameter is configured for the first model. Specifically, the first parameter can be a hyperparameter. By configuring the first parameter for the first model, the training performance and training effect of the first model are increased, thereby improving the text classification effect of the trained first model.

[0066] For example, the hyperparameters are set by the user, and the user can adjust the first parameters of the first model according to the classification task performed by the first model.

[0067] Example 3

[0068] An embodiment of the present invention provides a text classification method, such as... Figure 3 As shown, text classification methods include:

[0069] Step 302: Train each of the multiple pre-selected classification models based on the second text data;

[0070] Step 304: Calculate the text classification parameters corresponding to each pre-selected classification model after training;

[0071] Step 306: Based on the parameter values ​​of the text classification parameters corresponding to each pre-selected classification model, determine the first model and multiple second models from the multiple pre-selected classification models;

[0072] Step 308: Train multiple second models based on the first text data, and determine multiple second output data of the trained multiple second models;

[0073] Step 310: Train the first model based on multiple second output data and first text data;

[0074] Step 312: Optimize the trained first model and compress the optimized first model.

[0075] Step 314: Perform text classification processing on the text to be classified based on the compressed first model.

[0076] In this embodiment, each pre-selected classification model is trained using the second text data, ensuring fairness in selection by training each model on a single text dataset. This also facilitates the statistical analysis of the text classification parameters for each pre-selected model. By using the text classification parameters corresponding to each pre-selected model, the classification performance of each model can be more intuitively determined, increasing the accuracy of the assessment. Furthermore, by using the text classification parameter values ​​as the selection criteria for the first model and multiple second models, the classification performance of the first model and multiple second models is guaranteed, thereby enhancing the training effect of the second models on the first model.

[0077] In any of the above embodiments, the text classification parameters include at least one or a combination of the following: accuracy, precision, recall, and F1 score.

[0078] In this embodiment, the text classification parameters include one or more of accuracy, precision, recall, and F1 score. Accuracy represents the proportion of correctly predicted results out of the total number of classified samples; precision represents the proportion of actually positive samples among those predicted as positive; recall represents the proportion of actually positive samples among those predicted as positive; and the F1 score is the harmonic mean of precision and recall, meaning it combines both the model's precision and recall. These four types of parameters represent the specific classification performance of the model.

[0079] For example, the second text data includes two second text sub-data sets. That is, in the embodiments of this application, each pre-selected classification model needs to be trained twice. The first training is used to select the first model and the second model, and the second training is used to verify the accuracy of the selection in the first training. Specifically, the text data used in the first training is the Tsinghua News Text Classification Dataset, which contains 70,000 data points and is divided into 10 categories: education, real estate, finance, games, technology, entertainment, current affairs, sports, home furnishing, and fashion. When training begins, the data in the dataset is first randomly shuffled and divided into training set, validation set, and test set according to a ratio of 0.7, 0.2, and 0.1. Each pre-selected classification model is trained in three rounds, with 500 data points trained in each round. At the end of each round of training, the text classification parameters of the pre-selected classification model in this round of training are counted. After the count is completed, the next round of training is performed. After three rounds of training, the current training ends. For each pre-selected classification model, the text classification parameters with the best classification effect in the three rounds of training are taken as the text classification parameters of the model in this training. To more clearly illustrate the classification performance of the pre-selected classification models, this training uses the text classification parameters from the FastText model (a text classification model released by Facebook) for reference. Since the FastText model is lightweight, this training only calculates its precision and recall. Subsequently, the text classification parameters of each pre-selected classification model and the reference FastText model are mapped one-to-one, and a comparison table of the text classification parameters of each pre-selected model during the first training is presented, as shown in Table 1.

[0080] Table 1

[0081] bert-wwm-ext-chinese 0.9791 0.9756 0.9673 0.9714 hfl / roberta-wwm-ext 0.9798 0.9535 0.9576 0.9555 xlnet-base-cased 0.8354 0.9196 0.8470 0.8818 gpt2-medium-en 0.9346 0.8910 0.8784 0.8846 ernie-1.0 0.9808 0.9950 0.9971 0.9960 ernie-gram-zh 0.9634 0.9690 0.9562 0.9626 fasttext 0.9109 0.9102

[0082] Among them, the bert-wwm-ext-chinese model, hfl / roberta-wwm-ext model, xlnet-base-cased model, gpt2-medium-en model, ernie-1.0 model, and ernie-gram-zh model are pre-selected classification models, and the fasttext model is the reference model. For the four text classification parameters of accuracy, precision, recall, and F1 score, the larger the parameter value, the better the classification effect.

[0083] The selection criteria for both the first and second models should satisfy the following: excellent text classification ability. The selection criteria between the first and second models should satisfy the following: the first and second models have different model structures, and their corresponding word segmenters reference different Chinese lexicons. Regarding the number of second models, the criteria should be met simultaneously: select as many second models as possible, within the limits of GPU (graphics processing unit) performance and training time.

[0084] According to the selection rules for the first and second models and the text classification parameters of the pre-selected classification models, the number of second models is limited to three. The bert-wwm-ext-chinese model, hfl / roberta-wwm-ext model, ernie-1.0 model, and ernie-gram-zh model are selected as candidate models for the first and second models. Based on the model structure of the above four models, the bert-wwm-ext-chinese model is selected as the first model, and the hfl / roberta-wwm-ext model, ernie-1.0 model, and ernie-gram-zh model are selected as the second models.

[0085] After initially selecting the model, a second selection was conducted to verify its accuracy. The text data used in this second training iteration was the Chinese Scientific Literature Subject Classification Dataset, containing 5888 data entries across 67 categories. This dataset is characterized by a large number of categories and a small amount of data for each category. The training process was the same as the previous one, and will not be repeated here to avoid repetition. Table 2 shows a comparison of the text parameters after the second training.

[0086] Table 2

[0087] bert-wwm-ext-chinese 0.5956 0.8571 0.2727 0.4138 hfl / roberta-wwm-ext 0.6083 0.6667 0.2609 0.3750 xlnet-base-cased 0.1334 0.3636 0.2105 0.2667 gpt2-medium-en 0.0748 0.0 0.0 0.0 ernie-1.0 0.6287 0.8571 0.5714 0.6857 ernie-gram-zh 0.6194 0.3125 0.3846 0.3448 fasttext 0.4673 0.4673

[0088] As shown in Table 2, when the text data to be classified has many categories and a small amount of data for each category, the text classification ability of the xlnet-base-cased model and gpt2-medium-en model declines significantly. For the gpt2-medium-en model, it even becomes virtually unable to classify the text. While the bert-wwm-ext-chinese model, hfl / roberta-wwm-ext model, ernie-1.0 model, and ernie-gram-zh model also show a significant decrease in text classification performance, they still possess some classification ability for the text data. The second training session demonstrates the correctness of the selection of the first model and several second models.

[0089] In summary, the embodiments of this application use the bert-wwm-ext-chinese model as the first model; and the hfl / roberta-wwm-ext model, the ernie-1.0 model, and the ernie-gram-zh model as multiple second models.

[0090] For example, before selecting multiple second models, the Chinese scientific literature subject classification dataset is selected as the text to be classified. First, the first model before training is used to classify it, and the classification recall and F1 score of the first model before training are calculated. Then, the first model is trained based on multiple second output data and the first text data, and the first model after training is used to classify it again, and the classification recall and F1 score of the first model after training are calculated. The comparison table is shown in Table 3.

[0091] Table 3

[0092] The first model before training 0.273 0.414 The first model after training 0.696 0.762

[0093] As shown in Table 3, the classification recall of the first model after training increased by 154.9% compared with that before training, and the F1 score of the first model after training increased by 84.1% compared with that before training. Therefore, the above method can effectively improve the classification performance of the first model.

[0094] Example 4

[0095] An embodiment of the present invention provides a text classification method, such as... Figure 4 As shown, text classification methods include:

[0096] Step 402: Determine the first model and multiple second models from multiple pre-selected classification models based on the text classification parameters;

[0097] Step 404: Train multiple second models based on the first text data, and determine multiple second output data of the trained multiple second models;

[0098] Step 406: Train the first model based on multiple second output data and the first text data;

[0099] Step 408: Configure a first preset function for the first text data, and configure multiple second preset functions for multiple second output data;

[0100] Step 410: Configure a first weight for the first preset function and configure a second weight for the multiple second preset functions;

[0101] Step 412: Perform a weighted summation of the first preset function and multiple second preset functions according to the first weight and the second weight to obtain the first processing function;

[0102] Step 414: Optimize the first model according to the first processing function;

[0103] Step 416: Compress the optimized first model;

[0104] Step 418: Perform text classification processing on the text to be classified based on the compressed first model.

[0105] In this embodiment, firstly, different preset functions are configured for the first text data and multiple second output data. Specifically, a first preset function is configured for the first text data, and a second preset function is configured for the multiple second output data. Subsequently, different weights are configured for the different preset functions, and the preset functions with configured weights are weighted and summed to obtain the final first processing function. The first processing function is then used to optimize the first model.

[0106] Specifically, both the first and second preset functions are loss functions. By configuring different loss functions for the first text data and multiple second output data, different weights are assigned to different loss functions, and a weighted sum is performed based on the weights of the loss functions themselves. The weighted sum is then used as the loss function of the first model. This avoids the gradient explosion phenomenon caused by the continuous accumulation of error gradients during the training of the first model, ensuring the stability of the first model during operation, thereby improving the training effect and the model performance of the first model, and thus achieving the optimization of the first model.

[0107] For example, the first preset function can be the cross-entropy loss function; the second preset function can be the Kullback-Leibler divergence loss function; the first weight is 0.4, and the second weight is 0.2, meaning the ratio between the cross-entropy loss function and the Kullback-Leibler divergence loss function is 0.4:0.2:0.2:0.2; the specific values ​​of the first and second weights are set by the user based on the text data to be classified, and the first text data corresponds to the second output data of each second model. After determining the first processing function, it can also be divided by a constant N, where N represents the training degree of the second model on the first model. The larger the value of N, the deeper the training degree of the second model on the first model. Dividing the first processing function by the constant N can improve the optimization ability of the first processing function on the first model.

[0108] Example 5

[0109] An embodiment of the present invention provides a text classification method, such as... Figure 5 As shown, text classification methods include:

[0110] Step 502: Determine the first model and multiple second models from multiple pre-selected classification models based on the text classification parameters;

[0111] Step 504: Configure a different database for each of the multiple second models;

[0112] Step 506: Train multiple second models based on the first text data, and determine multiple second output data of the trained multiple second models;

[0113] Step 508: Train the first model based on multiple second output data and first text data;

[0114] Step 510: Optimize the trained first model and compress the optimized first model.

[0115] Step 512: Perform text classification processing on the text to be classified based on the compressed first model.

[0116] In this embodiment, after determining the first model and multiple second models, a different database is configured for each second model. Specifically, the database can be a lexicon. The lexicon is used by the word segmenter corresponding to the second model. When different lexicons are used, the vocabulary size and vocabulary range of the second model are also different. Therefore, selecting different vocabulary sizes for each second model allows the second model to have different vocabulary ranges, thereby enabling the first model to learn a wider range of vocabulary during training, increasing the training effect of the first model and the word segmentation effect of the first model after training, and thus improving the first model's ability to learn different words.

[0117] Example 6

[0118] An embodiment of the present invention provides a text classification method, such as... Figure 6 As shown, text classification methods include:

[0119] Step 602: Determine the first model and multiple second models from multiple pre-selected classification models based on the text classification parameters;

[0120] Step 604: Train multiple second models based on the first text data, and determine multiple second output data of the trained multiple second models;

[0121] Step 606: Train the first model based on multiple second output data and first text data;

[0122] Step 608: Optimize the first trained model;

[0123] Step 610: Perform knowledge distillation on the optimized first model;

[0124] Step 612: Adaptive processing is performed on the first model after knowledge distillation;

[0125] Step 614: Perform text classification processing on the text to be classified based on the compressed first model.

[0126] In this embodiment, firstly, the optimized first model undergoes knowledge distillation, thereby reducing the number of parameters in the model itself without losing the information obtained through training. This improves the classification speed of the model in specific classification scenarios while reducing resource consumption when performing text classification tasks. By adaptively processing the knowledge-distilled first model, the model structure can be adjusted according to the different requirements of classification accuracy and classification speed for different classification tasks, thereby improving the user experience.

[0127] For example, in the embodiments of this application, the first model adopts a knowledge distillation method that can be self-supervised distillation. That is, the first model completes a knowledge distillation process through its own training. Compared with the traditional method of knowledge distillation through a teacher model, it does not require pre-training of the teacher model, thus achieving the technical effect of distillation during training and reducing training time.

[0128] For example, the QQP dataset of the Multi-Task Natural Language Understanding and Analytics Platform (GLUE) benchmark was used to train the first model before and after compression for text classification. The classification accuracy and F1 score of the model were obtained, and the parameter size of the first model before and after compression was also obtained. The comparison table of the text classification parameters of the first model before and after compression is shown in Table 4, and the comparison table of the model size of the first model before and after compression is shown in Table 5.

[0129] Table 4

[0130] The first model before compression 0.9047 0.8751 The first compressed model 0.9053 0.8757

[0131] Table 5

[0132] The first model before compression 83M The first compressed model 55M

[0133] As shown in Tables 4 and 5, compared with the first model before compression, the first model before compression has 33.7% fewer model parameters, 6% higher classification accuracy, and 6% higher F1 score. Therefore, the first model after compression improves the text classification effect while reducing the resource consumption when performing text classification tasks.

[0134] Example 7

[0135] An embodiment of the present invention provides a text classification method in a specific application scenario, such as... Figure 7 As shown, both the model training service and the text classification service are set up in the text training and classification service platform 704. The text training and classification service platform 704 will determine whether to provide model training service or text classification service based on the operation of the user terminal 700.

[0136] Specifically, after the user terminal 700 selects the first model and configures the hyperparameters for the first model, the network service platform 702 immediately sends this information to the text training and classification service platform 704. After receiving this information, the text training and classification service platform 704 trains the second model in the text training and classification service platform 704, and guides the training of the first model based on the output data of the training text according to the trained second model. Subsequently, the trained first model is compressed to finally obtain the compressed first model, which is the first model used to classify the text to be classified.

[0137] When the user terminal 700 selects to input the text to be classified, the network service platform 702 sends this information to the text training and classification service platform 704. After receiving this information, the text training and classification service platform 704 loads the first model that has been trained on the text training and classification service platform 704 to classify the text to be classified and outputs the classification result.

[0138] In the embodiments of the present invention, the training of the first model and the classification of the text to be classified are realized through the text training and classification service platform 704, which reduces the workload of manually selecting the second model and training the first model, and improves the user experience.

[0139] Example 8

[0140] Another embodiment of the present invention provides a text classification device 800. Figure 8 A structural block diagram of a text classification apparatus 800 according to an embodiment of this application is shown, such as... Figure 8 As shown, the text classification device 800 includes: a determining unit 802, configured to determine a first model and a plurality of second models from a plurality of pre-selected classification models based on text classification parameters; a processing unit 804, configured to train the plurality of second models based on first text data, and determine a plurality of output data of the trained plurality of second models; the processing unit 804 is further configured to: train the first model based on the plurality of output data and the first text data; optimize the trained first model, and compress the optimized first model; perform text classification processing on the text to be classified based on the compressed first model; wherein, the optimization processing is used to improve the training degree of the first model.

[0141] The text classification apparatus 800 provided in this application includes a determining unit 802 and a processing unit 804. The determining unit 802 is used to determine a first model and multiple second models from multiple pre-selected classification models based on text classification parameters. When the determining unit 802 determines the first model and multiple second models, the processing unit 804 first trains the multiple second models using first text data to obtain multiple output data corresponding to the first text data. Then, it trains the first model based on the multiple output data and the first text data, and optimizes the trained first model to enhance its training level. Subsequently, it compresses the optimized first model to reduce the number of parameters, and uses the compressed first model as a text classification model to classify the text data in the text classification task.

[0142] Specifically, the first model can be a student model, and the second model can be a teacher model. The text classification method provided by this invention selects the second model through text classification parameters, ensuring the classification effect of the selected second model and thus better training the first model. The first model, trained with multiple second models, learns from their classification experience without needing extensive retraining, improving its text classification ability. This allows it to maintain excellent classification performance even with a small number of texts and many text categories, and ensures consistent excellent classification performance across different evaluation metrics. Optimizing the trained first model improves its text classification, reducing the probability of classification errors and further enhancing classification effectiveness. Compressing the optimized first model reduces the number of parameters, avoiding redundancy in specific classification scenarios. This improves classification speed while reducing resource consumption during text classification tasks, thus enhancing the user experience.

[0143] For example, in the embodiments of this application, the selection of the pre-selected classification model should meet the requirement that the selected pre-selected classification model has text classification function. Based on the above conditions, the embodiments of this application use the bert-wwm-ext-chinese model, hfl / roberta-wwm-ext model, xlnet-base-cased model, gpt2-medium-en model, ernie-1.0 model and ernie-gram-zh model as pre-selected classification models.

[0144] Example 8

[0145] Another embodiment of the present invention provides a text classification system, which includes a memory storing a program; and a processor that executes the program to implement the text classification method as described in any of the above embodiments.

[0146] The text classification system provided in this application includes a memory and a processor. The memory stores a program; the processor is connected to the memory and executes the program, implementing the text classification method as described in any of the above embodiments. Therefore, it possesses all the beneficial effects of the text classification method as described in any of the above technical solutions, which will not be elaborated further here.

[0147] Example 9

[0148] Another embodiment of the present invention provides a readable storage medium on which a program is stored, which, when executed by a processor, implements the text classification method as described in any of the above technical solutions.

[0149] The readable storage medium provided in this application embodiment is used to store a program that implements the text classification method as described in any one of the above technical solutions when executed by a processor. Therefore, it has all the beneficial effects of the text classification method as described in any one of the above technical solutions, which will not be repeated here.

[0150] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance, unless otherwise expressly specified and limited. The terms "connection," "installation," and "fixing," etc., should be interpreted broadly. For example, "connection" can mean a fixed connection, a detachable connection, or an integral connection; it can mean a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0151] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A text classification method, characterized in that, include: Based on text classification parameters, a first model and multiple second models are determined from multiple pre-selected classification models; The plurality of second models are trained based on the first text data, and the plurality of second output data of the trained plurality of second models are determined; The first model is trained based on multiple sets of second output data and the first text data; The first model after training is optimized, and the optimized first model is compressed. The compressed first model is used to perform text classification processing on the text to be classified. The optimization process is used to improve the training level of the first model; The step of determining the first model and multiple second models from multiple pre-selected classification models based on text classification parameters specifically includes: Train each of the plurality of pre-selected classification models based on the second text data; Calculate the text classification parameters corresponding to each trained pre-selected classification model; Based on the parameter values ​​of the text classification parameters corresponding to each pre-selected classification model, the first model and the multiple second models are determined from the plurality of pre-selected classification models.

2. The text classification method according to claim 1, characterized in that, Before training the plurality of second models based on the first text data, the method further includes: Configure the first model with a first parameter so that the first model is trained based on the first parameter, the plurality of output data and the first text data.

3. The text classification method according to claim 1, characterized in that, The text classification parameters include at least one of the following or a combination thereof: accuracy, precision, recall, and F1 score.

4. The text classification method according to any one of claims 1 to 3, characterized in that, The optimization process for the trained first model specifically includes: Configure a first preset function for the first text data, and configure multiple second preset functions for multiple second output data; A first weight is configured for the first preset function, and a second weight is configured for the plurality of second preset functions; The first preset function and the plurality of second preset functions are weighted and summed according to the first weight and the second weight to obtain the first processing function; The first model is optimized according to the first processing function.

5. The text classification method according to any one of claims 1 to 3, characterized in that, After determining the first model and multiple second models from multiple pre-selected classification models based on text classification parameters, the method further includes: Configure a different database for each of the plurality of second models.

6. The text classification method according to any one of claims 1 to 3, characterized in that, The compression process for the optimized first model specifically includes: The optimized first model is then subjected to knowledge distillation. The first model after knowledge distillation is then subjected to adaptive processing.

7. A text classification device, characterized in that, include: The determining unit is used to determine a first model and multiple second models from multiple pre-selected classification models based on text classification parameters; The processing unit is configured to train the plurality of second models based on the first text data and determine the plurality of output data of the trained plurality of second models; The processing unit is also used for: The first model is trained based on the multiple output data and the first text data; The first model after training is optimized, and the optimized first model is compressed. The compressed first model is used to perform text classification processing on the text to be classified. The optimization process is used to improve the training level of the first model; The determining unit is specifically used for: Train each of the plurality of pre-selected classification models based on the second text data; Calculate the text classification parameters corresponding to each trained pre-selected classification model; Based on the parameter values ​​of the text classification parameters corresponding to each pre-selected classification model, the first model and the multiple second models are determined from the plurality of pre-selected classification models.

8. A text classification system, characterized in that, include: A memory, wherein the memory stores a program; A processor that, when executing the program, implements the text classification method as described in any one of claims 1 to 6.

9. A readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the text classification method as described in any one of claims 1 to 6.

Images