Training Corpus Selection Method Based on the Results of Text Classification Models Using Multiple Loss Theory

Abstract

This application provides a method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions. At the model level, by fusing multiple loss functions, the text classification model adaptively learns the weights of each loss function on the classification performance, thereby improving its robustness. At the data level, based on the results of the text classification model fusion using the aforementioned multiple loss functions, variance calculation is used to determine the quality of the output category division of the training sample data, identifying poor-quality data for review and processing. The text classification model is then retrained based on the processing results to improve its classification or prediction performance. By calculating the confusion between output categories, a quantitative score is generated for the classification system of the text classification model, serving as the basis for adjusting the classification definitions within the model, thereby improving its prediction performance.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method for selecting training corpus based on the results of a text classification model that integrates multiple loss functions. Background Technology

[0002] In the data-driven era of artificial intelligence, human-computer dialogue has been widely used. In some application scenarios, the operation of smart terminals can be controlled by voice. The underlying basis of these operations is text classification, relying on text classification models.

[0003] Text classification models are trained using deep learning neural networks. For deep learning neural networks, obtaining high-quality training sample data ensures more accurate prediction results. Text classification models can classify and label large amounts of training sample data according to a certain classification system or standard. By learning from and training existing training sample data, they extract semantic features of the text in the training sample data to classify it and predict the output category.

[0004] In one implementation, the training sample data is unfiltered and contains a large amount of low-quality data. Since the text classification model depends on the quality of the training sample data during the training process, this low-quality and unfiltered data will inevitably affect the training process and training effect of the text classification model, thus affecting the accuracy of the text classification model. Summary of the Invention

[0005] This application provides a method for selecting training corpus based on the results of a text classification model that integrates multiple loss functions, in order to solve the problem of poor prediction performance of current text classification models.

[0006] Training corpus selection methods based on the results of text classification models that fuse multiple loss functions include:

[0007] Text classification models are divided into model level and data level according to their functions;

[0008] At the model level, multiple loss functions are fused to obtain the output category and category probability value of the training sample data in the text classification model.

[0009] At the data level, the variance of the training sample data is calculated based on the output category and the category probability value.

[0010] Select backtest data to be filtered out. The variance of the backtest data to be filtered out is lower than the variance threshold. The variance threshold is preset according to the output category of the training sample data itself.

[0011] As can be seen from the above technical solutions, at the model level of the text classification model, to address the problem of poor robustness, multiple loss functions are fused, allowing the text classification model to adaptively learn the weights of each loss function on the classification effect, thereby improving the robustness of the text classification model. At the data level of the text classification model, during the training process, based on the results of the text classification model fused with the above multiple loss functions, variance calculation is used to judge the quality of the output category division of the training sample data. The purpose is to identify the poor-quality data to be screened and back-checked, and to process the back-checked data. Based on the processing results, the text classification model is retrained to improve the classification or prediction effect of the text classification model. In addition, by calculating the confusion between output categories, a quantitative score is given to the classification system of the text classification model, which serves as the basis for adjusting the classification definition in the text classification model, thereby improving the prediction effect of the text classification model. In summary, the embodiments of this application, through the fusion of multiple loss functions and the screening of training corpora, achieve the improvement of the prediction effect of the text classification model from both the model level and the data level. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating the principle and logic of the text classification model in an embodiment of this application;

[0014] Figure 2 This is a schematic diagram of a dialogue system page of a terminal device in an embodiment of this application;

[0015] Figure 3 This is a schematic diagram illustrating the display effect of a terminal device when the text classification is poor in an embodiment of this application;

[0016] Figure 4 This is a schematic diagram of the overall framework for a method of selecting training corpus based on the results of a text classification model that integrates multiple loss functions.

[0017] Figure 5 This is a flowchart illustrating the process of selecting training corpus based on the results of a text classification model that integrates multiple loss functions.

[0018] Figure 6 A flowchart illustrating the process of fusing multiple loss functions in a text classification model to obtain the output category and category probability value;

[0019] Figure 7A flowchart illustrating the calculation of various loss functions for training sample data and a data flow diagram.

[0020] Figure 8 This is a schematic diagram illustrating the display effect of a terminal device after text classification enhancement in an embodiment of this application. Detailed Implementation

[0021] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0022] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0023] This application relates to, but is not limited to, text classification and dialogue systems. The dialogue system can be, for example, controlling a terminal device via voice commands. In one usage scenario, voice commands can be used to control televisions, air conditioners, etc., such as adjusting the air conditioner temperature or fan mode. Another example is using voice control to instruct the television to search for and play a specific category of program. This application does not specifically limit the scope of the dialogue system. Text classification is the underlying technical support for terminal display. For example, when searching for a specific category of program, such as "movies," on a television using a dialogue system, the effectiveness of the text classification directly affects the display effect of the dialogue system. If the underlying text classification is poor, the effect presented to the user will also be unsatisfactory.

[0024] Text classification is determined by the text classification model. Figure 1 This is a schematic diagram illustrating the principle and logic of the text classification model in an embodiment of this application, as shown below. Figure 1 As shown, text classification models can classify and label large amounts of training sample data according to a certain classification system or standard. After inputting training sample data (training corpus), the model learns from and trains on existing training sample data, extracting semantic features of the text in the training sample data, thereby classifying the training sample data and predicting the output category, i.e., outputting the final output category. Therefore, it is evident that text classification models play a decisive role in the effectiveness of text classification.

[0025] To intuitively demonstrate the relationship between underlying text classification and terminal device display effects, this application uses a smart TV as an example for illustrative purposes. It should be noted that in actual use cases, this is not limited to smart TVs; it is only used here as an example for ease of understanding. For instance, Figure 2This is a schematic diagram of a dialogue system page of a terminal device according to an embodiment of this application. In this embodiment, the display terminal of the smart TV is labeled as display device 200. Figure 2 As shown, the main page of the dialogue system can include media categories, such as entry options like "Movies," "Variety Shows," and "Music." When a user issues a voice control command through the dialogue system, the corresponding page effect will be displayed based on the selected category.

[0026] Text classification models are trained using deep learning neural networks, and they rely on training sample data, or training corpus. Training corpus refers to the language material used for training, which can be a collection of text resources of a certain quantity and scale. For example, it can be one or more training sample data (training corpus) collected or input by the terminal. High-quality training corpus can ensure more accurate prediction results for the text classification model, while low-quality training corpus will affect the final prediction results of the text classification model.

[0027] If the training corpus is unfiltered and contains a large amount of low-quality data, it will inevitably affect the training process and performance of the text classification model, thus impacting its accuracy. Therefore, to improve the prediction performance of the text classification model, it is necessary to improve the quality of the training corpus. This means filtering the training corpus, identifying and removing the low-quality data, and then, depending on the situation, performing secondary verification or re-labeling. The goal is to improve the quality of the training corpus and consequently, enhance the prediction performance of the text classification model.

[0028] Figure 3 This is a schematic diagram illustrating the display effect of a terminal device when text classification is poor in an embodiment of this application. For example, the voice command issued to the display device 200 through the dialogue system is {movies starring Andy Lau}, such as... Figure 3 As shown, the display device 200 outputs results including "Movie A," "Movie B," and "Movie C," but due to poor underlying text classification, results unrelated to the movie category but related to Andy Lau are also displayed, such as "Andy Lau Song 1," "Andy Lau Song 2," and "Andy Lau Talks About Movies," thus affecting the user experience. Therefore, underlying text classification is the foundation of terminal device display, and text classification is determined by the text classification model. The role of the text classification model is to predict the output category of the training sample data by learning and training from existing training sample data. Therefore, only by improving the classification and prediction performance of the text classification model can the quality of text classification be improved.

[0029] To improve the performance of the text classification model, this embodiment of the application can divide the text classification model into a model layer and a data layer. Figure 4This is a schematic diagram illustrating the overall framework of a training corpus selection method based on the results of a text classification model that integrates multiple loss functions. Figure 4 As shown, at the model level, to address the problem of poor robustness of current text classification models, based on experiments attempting several loss optimizations, this application improves the robustness of the text classification model by fusing multiple loss functions, that is, by deeply integrating different loss functions in the text classification model, allowing the text classification model to adaptively learn the weights of each loss function on the classification effect of the text classification model.

[0030] At the data level, during the training of the text classification model, based on the results of the text classification model fused with the aforementioned multiple loss functions, the training corpus in the text classification model results is screened. The purpose is to identify training corpora of poor quality, thereby improving the quality of the training corpus from a data perspective and enhancing the classification performance of the text classification model. Furthermore, based on the screening results of the training corpus, a quantitative score is generated for the classification system of the text classification model, serving as the basis for adjusting the classification definitions in the text classification model, thereby improving the prediction performance of the text classification model.

[0031] In this embodiment, the training corpus refers to the language material to be trained, which can be a collection of text resources of a certain quantity and scale. The size of the training corpus can be large or small; it can be one or a few training sample data points, or it can be a collective term for the collection of all training sample data to be trained. By optimizing the training sample data, the training corpus can be optimized simultaneously. Similarly, by optimizing the training corpus, the training sample data can be optimized simultaneously. The two complement each other and there is no strict boundary (they can also be the same, which is not limited in this application).

[0032] Specifically, see Figure 5 , Figure 5 This is a flowchart illustrating the training corpus selection method based on the results of a text classification model that integrates multiple loss functions. Figure 5 As shown, the method for selecting training corpus based on the results of a text classification model that integrates multiple loss functions includes the following steps:

[0033] S1: Divide the text classification model into the model layer and the data layer according to its functions;

[0034] From a functional perspective, text classification models can be divided into two distinct parts. One part is the data layer, which consists of training sample data, providing the data foundation for training the text classification model. The other part is the model layer, which is the text classification model itself, used to train the model using the training sample data.

[0035] S2: At the model level, multiple loss functions are fused to obtain the output category and category probability value of the training sample data in the text classification model.

[0036] In this embodiment, the fusion of multiple loss functions can be based on a fully connected neural network. A fully connected neural network is the most basic type of neural network, with the most network parameters and the largest computational cost. A fully connected neural network can be multi-layered, which can better improve the prediction effect and classification accuracy of the text classification model.

[0037] In one implementation, the training sample data is unfiltered, and most of the data is high-dimensional and sparse, resulting in weak text classification feature representation capabilities and inaccurate text classification. Fully connected neural networks, however, solve the text representation problem in large-scale text classification through deep learning. They transform text representation from a high-dimensional, sparse data type that is difficult for neural networks to handle into continuous, dense data similar to images and speech, enabling better feature extraction from text, labels, and related knowledge.

[0038] In one implementation, the text classification model uses the cross-entropy loss function and employs one-hot encoding for the data, meaning that only one state is labeled for each data category (hereinafter referred to as category). During the network learning process of one-hot encoded labels, the text classification model is encouraged to predict the probability of the target output category as close to 1 and the probability of the non-target output category as close to 0. That is, the probability of the target output category in the final predicted logits (in deep learning, logits can refer to the output of various classifiers, text classification models, or scores of a certain category) vector will tend to infinity. This causes the text classification model to learn in the direction of infinitely increasing logit difference between correctly and incorrectly predicted labels. However, an excessively large logit difference will make the model lack adaptability, leading to overconfidence in its prediction results and the problem of overfitting.

[0039] In one embodiment, to address the overfitting problem in text classification model computation, the cross-entropy loss function can be introduced or replaced with a label smoothing loss function. The label smoothing loss function is a regularization function that suppresses overfitting. It primarily uses soft one-hot encoding (soft labels) to better fit the actual scenario, reducing the weight of the output class of the real sample labels in the loss function calculation, ultimately suppressing overfitting. By replacing the cross-entropy loss function with the label smoothing loss function, the text classification model learns reasonably from the label probabilities.

[0040] In one embodiment, to address the issue of imbalanced samples between different output categories in text classification models, the cross-entropy loss function can be introduced or replaced with the Focal Loss loss function. The Focal Loss function is specifically designed to alleviate the imbalance between output categories and between easy and difficult samples in text classification models. This can be achieved by adding a hyperparameter to the cross-entropy. The hyperparameter can be derived by working backwards from the sample proportions of the training corpus, typically based on the inverse class frequency or verified using cross-entropy. To address the issue of different sample difficulty levels, a moderating factor can be introduced. This factor is less than 1, thus acting as a decay factor. For high-scoring samples, this factor has a rapid decay effect, and the closer it is to 1, the greater the decay. This allows the Focal Loss function to focus more on difficult classification samples.

[0041] In one embodiment, to improve the robustness and generalization of the text classification model (robustness refers to the text classification model's ability to resist interference, and generalization refers to the text classification model's adaptability to training data), the cross-entropy loss function can be introduced or replaced with an adversarial training loss function. Adversarial training is an important way to enhance the robustness of the text classification model during its learning process; poor robustness will affect the classification performance of the text classification model. For example, if a sample in the training dataset is "{How to make Chongqing noodles}", and one of the characters is changed to "{How to get Chongqing noodles}", the meaning of the sample after the change is actually the same as before, but the text classification model cannot classify it correctly. This is a manifestation of poor robustness in the text classification model.

[0042] Adversarial training loss functions provide a regularized learning algorithm that adds an acceptable maximum perturbation to the word vector layer of a text classification model, ensuring it maintains classification ability within its neighborhood. This introduces small fluctuations to the input vector, thereby improving the generalization ability (generalization refers to the machine learning algorithm's adaptability to new samples) and robustness of the text classification model. Furthermore, the adversarial perturbation in text classification does not act on the original discrete input, but rather on the continuous word embedding (in deep learning, embedding represents word embedding; word embedding maps / embeds sentences composed of words into a representation vector). For example, in adversarial training, let x represent the input and θ represent the model parameters. Adversarial training adds the following loss to the current loss function:

[0043]

[0044] Where x is the current input perturbation, These are the parameters of the current model, used as a constant in the loss (using...). Instead of θ indicating that the perturbation generation process should not be used for backpropagation gradients, in each adversarial training iteration, the perturbation generation process is first used to generate gradients for the current model. The most impactful disturbance r adv Then put r adv This is added to word embedding, and then the loss function above is minimized to make the text classification model more robust to perturbations, that is, to improve the text classification model's resistance to interference.

[0045] The above are just a few examples of feasible loss functions. In this embodiment, various loss functions may include, but are not limited to, the label smoothing loss function, the Focal Loss loss function, and the adversarial training loss function mentioned above. For example, multiple loss functions can be selected based on past experience or on experiments attempting several loss optimizations.

[0046] In this embodiment of the application, taking multiple loss functions including the aforementioned label smoothing loss function, Focal Loss loss function and adversarial training loss function as an example, the fusion of multiple loss functions refers to the fusion of the label smoothing loss function, Focal Loss loss function and adversarial training loss function according to a preset fusion method. Figure 6 This is a flowchart illustrating the process of fusing multiple loss functions to obtain the output category and category probability value for a text classification model, as shown below. Figure 6 As shown, the steps for fusing multiple loss functions to obtain the output category and category probability value in a text classification model may include:

[0047] S21: Input the training sample data into the text classification model to obtain the process category and process probability value;

[0048] The label smoothing loss function, Focal Loss loss function, and adversarial training loss function can all calculate the loss value of a batch of data in the current training sample data (batch data can refer to a portion of the training sample data input into the text classification model when calculating the loss value). Figure 7 A flowchart illustrating the calculation of various loss functions for training sample data, and a data flow diagram, such as... Figure 7 As shown, training sample data can be input into the text classification model via embedding. The text classification model will output predicted logits. The predicted logits can include process category and process probability value, but these logits are the result without loss fusion and do not represent the final output, so they are referred to as process category and process probability value.

[0049] S22: Calculate the process loss value of each loss function based on the process category and process probability value. After obtaining the process category and process probability value output by the text classification model according to step S21, the text classification model will calculate all loss types and obtain the process loss value of each loss function.

[0050] S23: Input the process loss value into the fully connected neural network to obtain the final loss value;

[0051] After calculating the process loss values ​​for each loss function, the text classification model outputs them to the next layer of the network. The next layer is a fully connected neural network. The results of the three loss functions are fed into the fully connected neural network, which then deeply fuses the loss values ​​of multiple loss functions, enabling the text classification model to adaptively learn different losses.

[0052] In practice, the text classification model can learn parameters on its own throughout the training process, enabling it to better adapt to the training sample data and improve its generalization ability in more scenarios. For example, the text classification model can learn how to choose between three loss functions to improve its prediction performance.

[0053] The calculation of the process loss values ​​can be achieved in the following ways: The text classification model can include a fusion module, which selects one or more loss functions to calculate the process loss values ​​of the text classification model. The fusion module can select the following loss functions:

[0054] In the first embodiment, the fusion module randomly selects the loss function for the current batch of data, so the process loss value of any loss function may be the final loss value. Since different loss functions have their own advantages, different loss functions can optimize the text classification model from multiple perspectives. In large-scale batch data, randomly selecting a loss function can also optimize the text classification model from different perspectives.

[0055] In the second embodiment, the fusion module calculates the process loss values ​​of all loss functions for the current batch of data, selects the process loss value with the largest result as the final loss value, and updates the relevant parameters of the text classification model using the loss function corresponding to the final loss value. For example, the process loss values ​​of the label smoothing loss function, the Focal Loss loss function, and the adversarial training loss function are A1 (label smoothing loss function), A2 (Focal Loss loss function), and A3 (adversarial training loss function), respectively. Where A1 > A2 > A3, and A1 is the maximum value among the process loss values, then A1 is the final loss value. The loss function corresponding to A1 is the label smoothing loss function, so the classification module updates the gradient descent parameters of the text classification model based on the process loss value of the label smoothing loss function. Typically, text classification models often exhibit non-convergence and gradient vanishing phenomena in the later stages of training; selecting the largest process loss value for gradient descent can alleviate this situation.

[0056] In the third embodiment, the fusion module sums the process loss values ​​of the three loss functions and takes the sum as the final loss value. The classification module then updates the gradient descent parameters of the text classification model based on the summed final loss value. This integrates the advantages of the three loss functions, fusing the loss value of the text classification model from multiple perspectives. For example, the label smoothing loss function can solve the overfitting problem during the text classification model's calculation; the Focal Loss loss function can solve the problem of imbalanced samples between different output categories in the text classification model; and the adversarial training loss function can improve the robustness and generalization of the text classification model. This allows for multi-faceted optimization of the text classification model, improving its prediction performance.

[0057] In the fourth embodiment, after summing the process loss values ​​of the three loss functions, the text classification model can add fully connected layers and output layers according to the actual training needs. The parameters of the fully connected layers can be learned from the batch data to adjust the proportion of each loss function in the total loss. This allows the text classification model to better adapt to the training sample data and different usage scenarios.

[0058] It should be noted that in actual training, text classification models may employ other methods for calculating the loss function and final loss value. The above are only four feasible methods and do not represent all possibilities. When training text classification models on training sample data, depending on the scenario and task, training is typically performed ten to dozens of times. Generally, in the first few training iterations, the text classification model will choose the third or fourth implementation method for selecting the loss function and final loss value. Subsequent training will generally randomly select the loss function and the method for calculating the final loss value to meet the generalization requirements of the text classification model. Finally, gradient descent is performed based on the largest process loss value of the current batch of data until the text classification model completes all training iterations.

[0059] S24: Train a text classification model based on the final loss value to obtain the output category and category probability value.

[0060] In step S23, regardless of which loss function the text classification model chooses as the final loss value, the text classification model will be trained based on this final loss value, such as updating relevant parameters of the text classification model. By selecting different loss functions and calculating the final loss value, the text classification model fully considers the weights of each loss during training, thereby improving the robustness of the text classification model at the model level.

[0061] To further understand the calculation process of various loss functions and the data flow relationships in text classification models, please refer to [link to relevant documentation]. Figure 7 Before any operations were performed, the initial training sample data consisted of "songs sung by Xiaohua on the Singer program, and movies / TV shows" (in actual processing, multiple training sample data will be processed simultaneously; this example only uses one data point to illustrate the data flow). The initial training sample data was in the form of a text sentence with a label; the text was "songs sung by Xiaohua on the Singer program," and the label was "movies / TV shows."

[0062] However, this type of labeling is a hard label, either 0 or 1, with only one output category and no other. It neither analyzes the text nor classifies and labels potential other output categories. The initial training sample data is processed by a text classification model based on multiple loss functions. The commonly used cross-entropy loss function is optimized into multiple loss functions to train the text classification model, and the hard labeling mode for the training sample data is abandoned. The text classification model outputs the results of the training sample data in the form of soft labels: "output category plus category probability value". For example, the initial training sample data "Songs sung by Xiaohua on the Singer program, Film and Television" outputs {"Film and Television":"0.60","Music":"0.40"} after being processed by a text classification model with multiple loss functions. Here, "Film and Television" and "Music" are the output categories, and "0.60" and "0.40" are the corresponding category probabilities.

[0063] It should be noted that at the model level, regardless of whether the text classification model is fused with multiple loss functions, the output of the text classification model is always a soft label in the form of "output category + category probability value". However, the difference is that a text classification model fused with multiple loss functions can perform better, improve the robustness and generalization of the text classification model, and thus improve the prediction and classification effect of the text classification model.

[0064] S3: At the data level, calculate the variance of the training sample data based on the output category and the category probability value.

[0065] Typically, the last layer in a text classification model is a softmax layer, also known as a logistic regression layer. Before the softmax layer, the output of a text classification model is not a probability; it is normalized to a probability value that sums to 1, for example, {"movies":"0.998","music":"0.0019"}. At the data level, based on the "output category + category probability value" result fused from the various loss functions mentioned above, the variance of the training sample data is calculated from this output category and category probability value.

[0066] The ideal output of the normalized text classification model is {"Film":"1.0","Music":"0.0"}, indicating that the text classification model is 100% certain about the classification result of the training sample data. If the output of the text classification model is {"Film":"0.50","Music":"0.50"}, it means that the text classification model cannot determine the classification result of the training sample data. This may be due to incorrect labeling of the training sample data, or the training sample data itself may be classified as either "Film" or "Music". Based on this, the embodiments of this application introduce variance to characterize the confidence level of a training sample data.

[0067] In statistics, variance is a measure of the dispersion of a set of data. In this embodiment, variance is used to determine the quality of the output class classification of the training sample data. The higher the variance, the higher the discrimination of the text classification model on the training sample data, and the better the quality of the output class classification of the training sample data; the lower the variance, the more difficult it is for the text classification model to distinguish the output class of the training sample data, and the relatively poor quality of the output class classification of these low-variable training sample data.

[0068] In one embodiment, the variance of the training sample data is calculated using the following formula:

[0069]

[0070] Where x is the sample value of the training sample data, μ is the mean of all sample values, and n is the number of samples. For example, if the output of the text classification model is {movies 0.4, music 0.3, games 0.2, food 0.1}, then in the above variance formula, x is 0.4, 0.3, 0.2, and 0.1 respectively, μ is 0.25, and n is 4.

[0071] S4: Filter out the back-lookup data to be filtered. The variance of the back-lookup data to be filtered is lower than the variance threshold. The variance threshold is preset according to the output category of the training sample data itself.

[0072] Considering that the variance distribution of training sample data differs across output categories, one embodiment uses a pre-set variance threshold to evaluate the quality of training sample data for each output category. For example, if the variance threshold for a certain output category is 'a', and the variance of a certain training sample data is 'b', if b is less than a, the training sample data is considered to have poor category classification quality and is considered data to be reviewed and requires secondary review. If b is greater than a, the training sample data is considered to have good category classification quality and does not require review. In this way, by calculating the variance of the training sample data and comparing it with the pre-set variance threshold, training sample data with potential labeling errors and possible classification difficulties can be quickly selected and reviewed, avoiding the need to re-examine all training corpora / training sample data.

[0073] It should be noted that after the data to be screened and reviewed is determined, it needs to undergo review processing. The method of review processing can be determined according to the actual situation, and this application does not impose specific limitations. For example, the corpus can be optimized or the training sample data can be re-labeled, and the text classification model can be retrained based on the processing results. When retraining, the data basis for training is no longer the low-quality data to be screened and reviewed, but the data after review processing. That is, the remaining data are all high-quality training sample data, which can enable the text classification model to have better classification or prediction performance.

[0074] Based on the filtering results of the back-search data to be filtered, this embodiment of the application further includes generating a question sample corresponding to each output category. The question sample contains all the back-search data to be filtered in the output category. For each output category, back-search data to be filtered is selected based on the variance threshold of that output category. All back-search data to be filtered under that output category are summarized to obtain the question sample corresponding to that output category. After obtaining the question sample corresponding to the output category, label pairs of the back-search data to be filtered are generated based on the question sample. All back-search data to be filtered with the same label pair are grouped into the same label pair class, which includes two output categories. Then, the confusion degree between the two output categories in the label pair class is calculated. If the confusion degree is lower than the confusion degree threshold, there is overlap between the output categories in the label pair.

[0075] Specifically, text classification models often use millions of training samples. When defining the output categories, unclear or confusing definitions are inevitable, affecting the model's performance. For example, a training sample might be "{Are there any tutorials on Java?}", which could be classified as either "{Movies}" or "{Adult Education}", making it difficult for the model to distinguish between them. Therefore, this application, in addition to evaluating the quality of the training sample data, also provides a quantitative score for the current classification system to determine whether adjustments are needed.

[0076] The initial classification of a training sample data point needs to be manually defined. For example, if the training sample data contains movie and actor names, it is manually classified as a movie / film output; if it contains music names, it is manually classified as a music output. Ideally, the boundaries between output categories are clear, such as movie / film being movie / film and music being music. However, in reality, especially when there are many output categories, the boundaries between different output categories may overlap and become unclear. Therefore, when there is a dispute over the output category, a value is needed to determine whether there is overlap between two or more output categories, and then the output category classification is adjusted to make the distinctions between output categories clearer.

[0077] In one implementation, the "confusion factor" is used to describe the classification status of the entire classification system, determining whether there is overlap between two or more output categories. One way to calculate the confusion factor is to generate label pairs for the back-search data to be filtered based on the problem samples, group all back-search data with the same label pairs into the same label pair class, which can include two output categories, and calculate the confusion factor between the two output categories in the label pair based on the categorized back-search data.

[0078] In specific implementation, the steps of generating label pairs for the back-search data to be filtered based on the problem samples include:

[0079] First, the problem sample includes the data to be filtered and sorted according to a preset sorting method. For example, the data can be sorted according to the magnitude of the output category probability value. Taking a certain data as an example, the sorting method can be {Movies 0.4, Music 0.3, Games 0.2, Food 0.1}, with the output category probability value of the larger value appearing first and the output category probability value of the smaller value appearing last.

[0080] Secondly, after confirming the sorting method, the data to be filtered and retrieved from the first two output categories are paired into a label pair. For example, in the example above, the first two output categories of the data to be filtered and retrieved are paired into a label pair, which is {Film 0.4, Music 0.3}. Since 0.4 and 0.3 are very close in value, it indicates that the output categories of Film and Music overlap.

[0081] For all problem samples, tag pairs are generated following the two steps described above. Samples with the same tag pairs (i.e., data to be filtered and retrieved) are grouped into the same tag pair class. For example, some tag pairs are as follows: Tag pair 1 is {"Movies":"0.60","Music":"0.30"}, Tag pair 2 is {"Movies":"0.60","Food":"0.30"}, Tag pair 3 is {"Music":"0.60","Opera":"0.30"}, Tag pair 4 is {"Movies":"0.50","Music":"0.40"}, and Tag pair 5 is {"Movies":"0.40","Food":"0.40"}. Among these 5 tag pairs, tag pair 1 and tag pair 4 both include "Movies" and "Music", so they are grouped into class 1; tag pair 2 and tag pair 5 both include "Movies" and "Food", so they are grouped into the same tag pair class; tag pair 3 has no matching tags, so it has its own tag pair class.

[0082] After the tag pairs are categorized, the confusion level between the two output categories in each tag pair is calculated based on the categorized back-search data to be filtered. When determining the degree of overlap between two output categories, for example, to determine the overlap between the output categories of "film" and "music," all back-search data categorized as "film" and "music" should be selected. By calculating the confusion level between each pair of output categories, the overall confusion level within the classification system can be calculated.

[0083] In one implementation, the confusion between the two output categories is calculated using the following formula:

[0084]

[0085] Here, Score refers to the confusion score, C1 and C2 are the sum of the number of samples of the two output categories in the label pair, K is the number of all question samples in C1 and C2, and σ is the variance of the training sample data. This formula can be used to calculate the confusion between the first two output categories of the output category. The same method can be used to calculate the confusion between any two output categories.

[0086] In the steps described above, the confusion score between any two output categories has been obtained using the confusion calculation formula. To determine whether there is overlap between two output categories, a confusion threshold can be set. One way to set the confusion threshold is to take the average of all confusion scores as the threshold. When the confusion score of a label pair is significantly lower than the confusion threshold, it is considered that the definitions of the two output categories in that label pair overlap. If there is overlap in the output categories of the label pair, the output categories need to be redefined. For example, a new output category can be defined based on the actual situation, or the training sample data corpus can be reorganized.

[0087] From the perspective of the effect presented on the terminal device, the improvement of the classification and prediction effect of the text classification model will inevitably improve the text classification effect, thereby improving the display effect of the terminal device. Figure 8 This is a schematic diagram illustrating the display effect of a terminal device after text classification enhancement in this embodiment. The voice command sent to the display device 200 via the dialogue system is still {movies starring Andy Lau}, such as... Figure 8 As shown, the effects output by display device 200 include "Movie A", "Movie B", "Movie C"... "Movie J", and... Figure 3 In comparison, the improved text classification model yields highly accurate search results, eliminating interference from other output categories and thus enhancing the user experience.

[0088] As can be seen from the above technical solutions, at the model level of the text classification model, to address the problem of poor robustness in current text classification models, multiple loss functions are fused, allowing the text classification model to adaptively learn the weights of each loss function on the classification effect, thereby improving the robustness of the text classification model. At the data level of the text classification model, during the training process, based on the results of the text classification model fused with the above multiple loss functions, variance calculation is used to judge the class division quality of the training sample data. The purpose is to identify the poor-quality data to be screened and back-checked, and to process the back-checked data. Based on the processing results, the text classification model is retrained to improve the classification or prediction effect of the text classification model. In addition, by calculating the confusion between categories, a quantitative score is given to the classification system of the text classification model, which serves as the basis for adjusting the classification definition in the text classification model, thereby improving the prediction effect of the text classification model. In summary, the embodiments of this application, through the fusion of multiple loss functions and the screening of training corpora, achieve the improvement of the prediction effect of the text classification model from both the model level and the data level.

[0089] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A method for selecting training corpus based on the results of a text classification model that integrates multiple loss functions, characterized in that: include: Text classification models are divided into model level and data level according to their functions; At the model level, multiple loss functions are fused to obtain the output category and category probability value of the training sample data in the text classification model. At the data level, the variance of the training sample data is calculated based on the output category and the category probability value; Select backtest data to be filtered out. The variance of the backtest data to be filtered out is lower than the variance threshold. The variance threshold is preset according to the output category of the training sample data itself. Generate a question sample corresponding to each output category, wherein the question sample contains all the back-search data to be filtered in the output category; Generate label pairs for the back-search data to be filtered based on the problem sample, and group all back-search data with the same label pairs into the same label pair class, which includes two output categories; Calculate the obfuscation degree between the two output categories in the label pair class; The classification definition in the text classification model is adjusted based on the level of confusion.

2. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 1, characterized in that, The fusion of multiple loss functions is based on a fully connected neural network, and the fusion of multiple loss functions is to fuse the label smoothing loss function, the Focal Loss loss function, and the adversarial training loss function according to a preset fusion method.

3. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 2, characterized in that, The text classification model is fused using multiple loss functions to obtain the output category and category probability value, including: The training sample data is input into the text classification model to obtain the process category and process probability value; Calculate the process loss value of each loss function based on the process category and the process probability value; The process loss value is input into the fully connected neural network to obtain the final loss value; The text classification model is trained based on the final loss value to obtain the output category and category probability value.

4. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 2, characterized in that, The fully connected neural network is a multi-layer fully connected network.

5. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 1, characterized in that, Also includes: Once the data to be screened and retrieved is determined, the data to be screened and retrieved is processed, and the text classification model is retrained based on the processing results.

6. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 1, characterized in that, The variance of the training sample data is calculated using the following formula: = ； Where x is the class probability value of the training sample data. is the average of the class probabilities of all samples, and n is the number of samples.

7. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 1, characterized in that, The category probability values ​​are obtained after normalization.

8. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 1, characterized in that, Also includes: If the confusion level is lower than the confusion level threshold, there is an overlap of output categories in the label pair.

9. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 8, characterized in that, Generate the label pairs for the back-search data to be filtered based on the problem sample, including: The data to be filtered and retrieved is sorted according to a preset sorting method; The data to be filtered and retrieved from the first two output categories are combined into label pairs.

10. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 8, characterized in that, The confusion level between the two output categories is calculated using the following formula: Score = ; in, , These are the sum of the number of samples in the two output categories of the label pair, where K is... , The number of all problem samples in the dataset. Let V be the variance of the training sample data.

11. The method for selecting training corpus based on the results of a text classification model fusion using multiple loss functions as described in claim 8, characterized in that, It also includes redefining the output category if the label pair has an overlap with the output category.

Images