An unbalanced multi-label text classification method and an intelligent device

Abstract

The application provides an unbalanced multi-label text classification method and an intelligent device. After obtaining a text to be classified, the method can perform sampling in the text to be classified to obtain a sampling sample set. A text splicing sample set and a mixed enhancement sample set are constructed respectively. Finally, the text to be classified, the text splicing sample set and the mixed enhancement sample set are input into a training model for model training. The method can construct the text splicing sample set and the mixed enhancement sample set by splicing sample data and mixing the corresponding digital vectors of the sample data, expand the data volume of the sample data, and alleviate the problem of low accuracy caused by unbalanced sample data. Moreover, different loss functions are used to calculate the loss for different data sets, so as to update the model parameters of the training model through back propagation, thereby improving the accuracy of model calculation.

Description

Technical Field

[0001] This application relates to the field of text classification technology, and in particular to an imbalanced multi-label text classification method and intelligent device. Background Technology

[0002] Text classification is a data processing method in which computer and other data processing devices automatically classify and label text sets according to a certain classification system or standard. Text classification can be based on deep learning neural network technology, which finds the relationship model between text features and text categories in already labeled training sample data, and then uses the learned relationship model to determine the category of new text, thereby achieving semantic understanding of natural language text.

[0003] Before text classification, training samples are needed to train the model. These training samples are pre-labeled text data with classification labels. Typically, the text data corresponding to the training samples also comes from real-world business data. Since real-world business data is generated by specific business processes, which are often concentrated in a few key business areas, it exhibits a certain degree of imbalance. Therefore, the amount of data in each category in the sample data formed based on the labeled business data is also uneven, reducing the model's ability to cover all categories and affecting its accuracy.

[0004] To mitigate the impact of imbalanced training data on model accuracy, oversampling can be used for a minority of classes during model training to meet training requirements. However, oversampling simply repeats the same data for training, without actually increasing the amount of data for the minority class or reflecting the differences between the samples. Consequently, the improvement in training performance with fewer samples is not significant, and the model's accuracy is reduced. Summary of the Invention

[0005] This application provides an imbalanced multi-label text classification method and intelligent device to solve the problem of low accuracy caused by imbalanced sample data in traditional text classification methods.

[0006] On the one hand, this application provides an imbalanced multi-label text classification method, including the following steps:

[0007] Obtain the text to be classified, which is training sample data with multiple classification labels;

[0008] Sampling is performed on the text to be classified to obtain a sample set;

[0009] Construct a text splicing sample set, which is formed by splicing sample data from the sampling sample set;

[0010] Construct a hybrid augmented sample set, which is formed by mixing the numerical vectors corresponding to the sample data in the sampled sample set. The numerical vectors are formed by the transformation of the text to be classified through the network layer in the training model.

[0011] The text to be classified, the text concatenation sample set, and the hybrid augmentation sample set are input into the training model. The training model uses different loss functions to calculate the loss for the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set, so as to update the model parameters of the training model through backpropagation.

[0012] On the other hand, this application also provides a smart device, including: a storage module and a processing module, wherein the storage module is configured to store a training model; and the processing module is configured to execute the following program steps:

[0013] Obtain the text to be classified, which is training sample data with multiple classification labels;

[0014] Sampling is performed on the text to be classified to obtain a sample set;

[0015] Construct a text splicing sample set, which is formed by splicing sample data from the sampling sample set;

[0016] Construct a hybrid augmented sample set, which is formed by mixing the numerical vectors corresponding to the sample data in the sampled sample set. The numerical vectors are formed by the transformation of the text to be classified through the network layer in the training model.

[0017] The text to be classified, the text concatenation sample set, and the hybrid augmentation sample set are input into the training model. The training model uses different loss functions to calculate the loss for the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set, so as to update the model parameters of the training model through backpropagation.

[0018] As can be seen from the above technical solutions, the imbalanced multi-label text classification method and intelligent device provided in this application can obtain a sample set by sampling the text to be classified after acquiring it. A text concatenation sample set and a hybrid augmentation sample set are then constructed. Finally, the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set are input into the training model for model training. This method can construct the text concatenation sample set and the hybrid augmentation sample set by concatenating sample data and mixing the corresponding numerical vectors, thus expanding the amount of sample data and alleviating the problem of low accuracy caused by imbalanced sample data. Furthermore, different loss functions are used to calculate the loss for different datasets to update the model parameters of the training model through backpropagation, thereby improving the accuracy of model calculation. Attached Figure Description

[0019] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the natural language processing flow in the embodiments of this application;

[0021] Figure 2 This is a schematic diagram of intelligent dialogue interaction in the embodiments of this application;

[0022] Figure 3 This is a schematic diagram of the collaborative workflow between the smart device and the server in the embodiments of this application;

[0023] Figure 4 This is a schematic diagram of the network application working scenario in the embodiments of this application;

[0024] Figure 5 This is a schematic diagram of the intelligent device structure in the embodiments of this application;

[0025] Figure 6 This is a schematic diagram illustrating the process of determining a minority sample set in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the sampling process in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram illustrating the process of constructing a text splicing sample set in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram illustrating the process of constructing a hybrid enhanced sample set in an embodiment of this application;

[0029] Figure 10This is a schematic diagram of the process for updating model parameters in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of the unbalanced multi-label text classification method in the embodiments of this application. Detailed Implementation

[0031] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0032] In this embodiment, the text classification method can be applied to smart devices with data processing capabilities and natural language processing requirements. These smart devices include, but are not limited to, computers, smart terminals, smart TVs, smart wearable devices, smart display devices, and servers. The smart device may have a built-in or external storage module and provide a processing module to form a text classification system capable of executing the text classification method.

[0033] For example, the smart device could be a smart TV, which has a built-in memory and controller. The memory can store data such as text, natural language processing models, and control programs. The controller can retrieve data from the memory and process the retrieved data by running the control program.

[0034] like Figure 1 As shown in this embodiment, natural language processing may include two stages: a model training stage and a text classification stage. In the model training stage, the controller can obtain training sample data, i.e., labeled text data, from the network or other means. The training sample data is then input into an initial training model for training. The training model can output classification probabilities for each classification label based on the input sample data. The output classification probabilities are then compared with the classification labels to obtain the error between the classification result and the classification label, and the error is backpropagated to adjust the model parameters in the training model. In this way, after a certain amount of training sample data, the parameters of the training model are repeatedly fine-tuned to obtain a training model with high classification accuracy. After completing the model training process, the controller stores the trained classification model in memory for subsequent application use.

[0035] During the text classification stage, the controller can retrieve the trained classification model from memory and input the text data to be classified into the retrieved model. Through the internal calculations of the classification model, the probability of the current text data being classified for each label can be obtained, thereby understanding the semantics of the text data.

[0036] The training model or classification model described in the above embodiments can be a Natural Language Processing (NLP) model. For example, the BERT model and other NLP models obtained by optimizing or modifying the BERT model. It should be noted that the model described in the above embodiments can be called a training model during the model training phase and a classification model during the text classification phase. Since the training model and the classification model are simply different stages of the same model, both the training model and the classification model take text data as input during text classification. Therefore, text data that can be input into the training model can also be input into the classification model. That is, in any of the embodiments described below, unless otherwise stated, the training model and the classification model will not be distinguished, and the text classification method described using the Natural Language Processing (or NLP) model will be uniformly used. This method can be applied to the processing of text data during the model training phase and also to the processing of text data during the text classification phase.

[0037] The text classification process described above allows smart devices to determine the probability of classifying text data according to different classification labels, thereby determining the meaning of the text data. In other words, the text classification process can determine machine language from natural language text data, enabling machine learning. Therefore, the text classification process can be applied to fields related to natural language processing, such as intelligent voice control, intelligent question answering, image recognition processing, and business statistical analysis.

[0038] In some embodiments, to implement the above-described text classification process, smart devices can train models and classify text by embedding artificial intelligence (AI) algorithms within the operating system. For example, ... Figure 2 As shown, an intelligent question-answering robot can have a built-in intelligent question-answering system in its operating system. In practical applications, the intelligent question-answering robot can acquire user-inputted text data in real time, such as "Where is store ×× located?" It then calls the intelligent question-answering system, inputting the user-inputted text data into a classification model within the system to understand the semantics of the user-inputted text data, i.e., "Search for ×× location". Finally, based on the understood semantics, it provides corresponding dialogue content, i.e., "Store ×× is on the third floor, 308F", thus realizing the intelligent question-answering function.

[0039] Clearly, for smart devices used for different purposes, the built-in artificial intelligence algorithms in their operating systems differ due to the varying functions they perform. However, they all essentially perform text classification processes, differing only in the setting of classification labels. For example, the classification labels set for the classification model in the AI ​​algorithm built into a smart question-and-answer robot are labels related to the question-and-answer process, such as "search," "consult," "select," and "suggestion" to indicate the user's intent. Furthermore, different user intents can apply to different business objects, so classification labels can also include business object names, such as "title," "item name," "person's name," and "place name." In contrast, the classification labels for the social classification model in the smart voice system built into a smart TV can be labels related to media asset playback, such as "movie," "TV series," and "animation" to indicate the media asset type, as well as "film title," "author," and "genre (comedy, military, rural)" to indicate the media asset object.

[0040] Besides embedding AI algorithms within the operating system, the AI ​​algorithms for text classification can also be built into applications. In some embodiments, smart devices can also implement text classification by installing an application. The application capable of text classification can be a system application or a third-party application. For example, to implement intelligent question-and-answer functionality, a computer can download and install an "intelligent question-and-answer robot" application, run the application to invoke a classification model, and then, by acquiring the user's input text data in real time and inputting the text data into the classification model, achieve the text data classification function.

[0041] In some embodiments, the text classification function is not limited to being implemented by a single smart device, but can also be implemented through the collaborative operation of multiple devices. That is, the smart device can establish a communication connection with the server. In practical applications, the smart device acquires the text data input by the user in real time, the server performs model training and text classification processes, and then the smart device displays the classification results.

[0042] For example, such as Figure 3 As shown, the smart device can acquire text data input by the user in real time during operation and send the text data to the server. The server has a built-in AI algorithm and classification model that implements text classification. After receiving the text data sent by the smart device, the server can input the text data into the classification model to obtain the classification result output by the model. The server then feeds back the classification result to the smart device, so that the smart device can provide feedback on the classification result and related interactive information to the user.

[0043] Clearly, to meet diverse business needs and reduce data processing volume, in practical applications, the specific device data for text classification through multi-device collaborative operation can be flexibly configured according to the requirements of the implemented function. Furthermore, the specific text classification process can be flexibly set according to device hardware configuration and data volume requirements to reduce repetitive data processing and conserve device computing power. For example, multiple smart devices can simultaneously establish communication connections with a server. The server provides a unified classification model for multiple smart devices. After obtaining the classification model, different smart devices can independently perform data input, model calculation, and result output to achieve text classification. Simultaneously, smart devices can also report their processed text data to the server for further model training and continuous improvement of the classification model. Therefore, the server can push classification models to smart devices at predetermined times, updating the classification models in each smart device to maintain their timeliness.

[0044] Furthermore, when text classification is implemented through multi-device collaborative operation, the computational load of each device can be monitored in real time, and the actual execution entity of the model training phase and / or text classification phase can be dynamically adjusted based on the real-time load. That is, Figure 4 As shown, in some embodiments, the application used to implement text classification can be a web application. Smart devices and servers connected to the same network can implement text classification by installing and running the web application. During the operation of the web application, it can monitor the computational load of each smart device and server in real time, including CPU usage, memory usage, and network latency. When any abnormality occurs in the data corresponding to any computational load, the AI ​​algorithm execution entity of the corresponding device can be adjusted in real time to ensure the smooth operation of the text classification process.

[0045] For example, under normal circumstances, model calculations in the text classification process can be performed by smart devices. However, when the memory usage of a smart device exceeds a threshold, the process of performing model calculations can be paused. The smart device can then be automatically controlled to send the acquired text data to the server, which will then perform model calculations and provide the classification results. This reduces the processing load on the smart device and improves the timeliness of text classification.

[0046] As can be seen from the above implementation methods, in the process of applying text classification, intelligent devices or servers need to input text data into the training model (or classification model) during the model training phase and the text classification phase. Since the text data to be classified is natural language text data, it has different text formats depending on its source. For example, for text data generated from user-inputted voice information, its content tends to be colloquial based on the user's voice input process, and the actual text length is relatively short, generally only one or a few sentences. However, for business texts such as contracts, judgments, and agreements, due to their specific format requirements, their text length is longer, generally including multiple paragraphs, each paragraph containing multiple sentences.

[0047] However, due to the inherent imbalance in the business itself, the corresponding business documents also exhibit imbalance, making data imbalance prone to occur during text classification. For example, in the film classification domain, because the number of comedy films is greater than that of art films, the training data for film classification will have more data labeled "comedy" than "art film." With imbalanced sample data, during model training, many samples are likely to be updated multiple times during the gradient descent phase, thus reducing the generality of the trained model and leading to inaccurate classification results.

[0048] To improve the problem of inaccurate classification results, some embodiments of this application provide an imbalanced multi-label text classification method. This method can be applied to intelligent devices with storage and processing modules to augment training samples and alleviate the problem of data imbalance. Figure 5 As shown, the imbalanced multi-label text classification method includes the following:

[0049] Obtain the text to be classified. In multi-label scenarios, the text to be classified is training sample data with multiple classification labels. Based on the frequency of each label in the text, the text to be classified includes a majority sample set and a minority sample set. During the model training phase of text classification, the smart device or server can first obtain the text to be classified. The method of obtaining the text to be classified can also differ depending on the smart device's functionality. For example, for a smart question-answering robot, the text to be classified can be obtained by collecting user voice data through a voice acquisition device and then converting it using a speech-to-text tool. For a computer performing auditing tasks, the text to be classified can be obtained by reading business documents stored in a database.

[0050] Since the text to be classified is used for model training, the text acquired by the smart device is training sample data with multiple classification labels. For example, for a business document text1, its classification category, i.e., label1, can be manually labeled according to the actual meaning expressed in the document. Because the actual content contained in the business document can include content related to multiple categories, a business document text1 can be assigned multiple classification labels, i.e., for a business document text1, the classification labels that can be labeled include label1, label2, ..., labeln.

[0051] Clearly, due to the imbalance in data volume, the training sample data with some popular classification labels is larger, while the training sample data with some less popular classification labels is smaller. In this embodiment, for ease of description, the training data with popular classification labels is referred to as the majority samples, and all majority samples can form the majority sample set. Similarly, the training data with less popular classification labels is referred to as the minority samples, and all minority samples can form the minority sample set.

[0052] To determine which data points in the training samples are majority samples and which are minority samples, such as... Figure 6 As shown, in some embodiments, after acquiring the text to be classified, the smart device can iterate through the number of samples in the text to be classified and the number of times each classification label appears in the text to be classified. For example, the number of samples in the text to be classified (training set) is N, and the number of times label i appears in the training set is N. i Each text in the training set is denoted as t. i Its label is denoted as l i,1 ...l i,n .

[0053] In some embodiments, the number of samples N and the number of occurrences N are traversed. i Then, the smart device can compare the occurrence frequency of each category label to determine if there is a data imbalance problem. For example, the smart device can extract the occurrence frequency N of the category label that appears most frequently from the traversed occurrence counts. max The number of occurrences N corresponding to the category label with the fewest occurrences min Then compare the maximum difference in data volume, that is, calculate N. max and N min The difference in data volume. If the maximum difference in data volume is greater than a preset threshold, it indicates that the current training data is imbalanced.

[0054] In some embodiments, the number of samples N and the number of occurrences N are traversed. i Then, the smart device can also calculate the probability P of each category label appearing. iWherein, the probability of occurrence is the number of times N corresponding to the category label appears. i The ratio to the sample size N, i.e.:

[0055]

[0056] The calculated sample types can be used to classify the sample data, i.e., based on the occurrence probability P. i The sample data in the text to be classified is divided into majority and minority sample sets. In some embodiments, the smart device may, when classifying sample types, first compare the occurrence probability with a sample judgment threshold, where the sample judgment threshold is the average occurrence probability of all classification labels, i.e.:

[0057]

[0058] If the probability of the current category label appearing is greater than or equal to the sample judgment threshold, that is... p i ≥ mean ( p Then determine the current category label. i The corresponding training sample data belongs to the majority sample set; similarly, if the occurrence of the current classification label is less than the sample judgment threshold, i.e. p i < mean ( p Then determine the current category label. i The corresponding training sample data belongs to a minority sample set.

[0059] Obviously, in practical applications, other data can also be used as the sample judgment threshold. For example, the sample judgment threshold can be determined based on the amount of training data required by the model; that is, if the minimum amount of training data required by the model is M, then the number of times the classification label appears is N. i When the number of training data points is less than the minimum training data size M, the training data corresponding to that classification label belongs to the minority sample set. Conversely, when the number of times the classification label appears N times... i When the minimum training data size M is greater than or equal to the minimum training data size, the training data corresponding to the classification label is determined to belong to the majority sample set.

[0060] For the minority sample set in the training samples, due to its small data volume, it is necessary to expand the sample capacity to increase the update opportunities for minority samples. That is, the intelligent device can perform sampling in the text to be classified to obtain a sampled sample set. For classification labels where pi < mean(p), since their occurrence probability is less than the average probability, they are considered as minority samples and need to be augmented. Since the methods for augmenting minority samples in this embodiment include text concatenation and mixup class enhancement, during the sampling process, different sizes of sampled sample sets can be sampled from the text to be classified according to the different dataset sizes generated by text concatenation and mixup class enhancement methods.

[0061] That is, Figure 7 As shown, in some embodiments, when the smart device performs sampling in the text to be classified, it first obtains a first efficiency parameter and a second efficiency parameter. The first efficiency parameter is used to calculate the number of samples in the sample set of the spliced ​​sample set; the second efficiency parameter is used to calculate the number of samples in the sample set corresponding to the hybrid augmented sample set. For example, the first efficiency parameter is θ, and the second efficiency parameter is γ.

[0062] After obtaining the first efficiency parameter and the second efficiency parameter, the intelligent device can calculate the sampling median value, wherein the sampling median value is the ratio of the average occurrence probability of all category labels to the occurrence probability of the current category label, that is:

[0063]

[0064] In the formula, p m This is the intermediate value of the sample; This represents the average probability of occurrence for all category labels. Category tags i The probability of its occurrence.

[0065] After calculating the median value, the smart device can determine a first number of samples to be taken from the text to be classified. This first number is equal to the product of a first efficiency parameter and the median sample value, i.e.:

[0066]

[0067] In the formula, Q 1 The first quantity; θ The first efficiency parameter; p m This is the intermediate value of the sample; This represents the average probability of occurrence for all category labels. Category tags i The probability of its occurrence.

[0068] Through the above steps, the intelligent device can sample a first number of data points from the text to be classified to generate a first sample set. Similarly, the intelligent device can also sample a second number of data points from the text to be classified based on a second efficiency parameter γ to generate a second sample set, where the second number is equal to the product of the second efficiency parameter and the intermediate sampling value, i.e.:

[0069]

[0070] In the formula, Q 2 The first quantity; γ This is the second efficiency parameter; p m This is the intermediate value of the sample; This represents the average probability of occurrence for all category labels. Category tags i The probability of its occurrence.

[0071] It should be noted that the first and second efficiency parameters mentioned above are preset hyperparameters. Furthermore, the first and second efficiency parameters can be set to the same value or different values. Since text concatenation and vector mixing produce different final results, in most cases, the first and second efficiency parameters need to be set to different specific values.

[0072] After obtaining the sample set through sampling, the smart device can construct a text concatenation sample set and a hybrid augmented sample set, respectively. The text concatenation sample set is formed by concatenating sample data from a minority sample set and the sampled sample set; the hybrid augmented sample set is formed by mixing the numeric vectors corresponding to the sample data from the minority sample set and the sampled sample set. Clearly, the sampled sample sets used are different in the process of obtaining augmented data through different methods. That is, in the step of constructing the text concatenation sample set, at least two sample data from the first sampled sample set are concatenated to construct the text concatenation sample set; in the step of constructing the hybrid augmented sample set, the numeric vectors corresponding to at least two sample data from the second sampled sample set are mixed to generate the hybrid augmented sample set.

[0073] For text concatenation methods, such as Figure 8As shown, in some embodiments, after obtaining a first sample set, the smart device can call a concatenation function to concatenate at least two sample data points from a minority sample set and the sampled sample set. The concatenation function is then used to perform text concatenation on the sampled sample set to obtain concatenated text; then, concatenation labels are sequentially set for each concatenated text to generate a concatenated sample set. The concatenation labels are the union of the classification labels corresponding to the concatenated sample data. Furthermore, to enable the model to recognize the training data obtained through text concatenation, the concatenated text also includes delimiters.

[0074] For example, after collecting a sample set of Q1 numbers from the text data to be classified, for each data sample < t i , l i > It can extract a single sample from the sample set. t j , l j >. Then, based on the data sample < t i , l i >and sampled samples< t j , l j > This allows us to construct new spliced ​​data, namely:

[0075]

[0076] in, This is a concatenation function; l i | l j It is the union of the data sample labels and the sampled sample labels.

[0077] As can be seen, based on the above data concatenation method, after concatenating the text ti and the text tj together, we can obtain:

[0078]

[0079] or:

[0080]

[0081] Here, "[sep]" represents the separator and can be used as a segmentation marker for the Next Sentence Prediction (NSP) task.

[0082] While performing text concatenation, the intelligent device can also mix the corresponding numerical vectors from the training data in the minority sample set and the sampled sample set to construct a hybrid augmented sample set. These numerical vectors are formed by transforming the text to be classified through the network layers in the training model. Figure 9 As shown, in some embodiments, for the hybrid enhancement process, the smart device can first input the training data from the text to be classified into the network layer, i.e., the embedding layer, in the training model, so that the embedding layer can perform vector mapping on the input training text to generate numerical vectors. After obtaining the numerical vectors output by the network layer, the smart device can set the mixing ratio parameter and, according to the mixing ratio parameter, perform the mixed-class enhancement mixup algorithm on the numerical vectors to generate a hybrid enhanced sample set.

[0083] The mixing ratio parameter is a pre-set hyperparameter that follows a uniform or Beta distribution. For example, for a data sample in a minority sample set... t i , l i > It can extract a single sample from the sample set. t j , l j > is used for mixup operations, so < needs to be determined t i , l i > and < t j , l j The mixup ratio is >. Therefore, assume the blend enhancement result obtained after the mixup operation is:

[0084]

[0085] in, For data samples < t i , l i The numerical vector obtained after transformation by the embedding layer; For sampled samples < t j , l j >The digital vector obtained after conversion by the embedding layer; μ is the mixing ratio parameter.

[0086] As can be seen, the mixing ratio parameter can be determined in several ways. For example, the mixing ratio parameter can be assumed. μ It satisfies a uniform distribution, that is:

[0087]

[0088] Here, b is a hyperparameter representing the lower limit of the uniform distribution. It can be preset according to the actual application; for example, b can be set to 1 or 2. Therefore, based on the uniform distribution, the intelligent device can determine the mixing ratio parameter corresponding to each category through optimization.

[0089] In some embodiments, the mixing ratio parameter may also conform to a Beta distribution, and Bayesian tools are used to optimize the Beta distribution to determine the mixing ratio parameter corresponding to each category of data. That is, the mixing ratio parameter μ satisfies:

[0090]

[0091] Since the upper bound of the Beta distribution is 1 and the lower bound is 0, a hyperparameter α can be added to the distribution function to determine the upper bound in order for the distribution to exceed the upper bound, i.e.:

[0092]

[0093] Here, α is a hyperparameter used to represent the maximum weight of the sampled data distribution. The hyperparameter α can be manually determined. Parameters a and b can also be initially set to equal 1, i.e., a uniform prior distribution of 0-1, and then optimized using Bayesian optimization during subsequent training.

[0094] As can be seen, in the process of determining the mixing ratio parameters described above, the mixing ratio parameter μ follows the same distribution when each category is mixed up. However, this is obviously not true. Therefore, in practical applications, it can be assumed that the mixing ratio parameter μ for each category follows a Beta distribution, and then the final mixing ratio parameter value can be determined through Bayesian optimization. It should be noted that the parameter for category k is... μ k And it satisfies:

[0095]

[0096] Then when μ 1 = μ 2 =…= μ k When =…, the mixing ratio parameter satisfies a uniform distribution.

[0097] It should be noted that in the process of constructing the text concatenation sample set and the hybrid enhancement sample set described above, the sample data participating in text concatenation and hybrid enhancement can come from the text to be classified, and optionally from a minority sample set and a sampled sample set. For example, data samples < t i , l i >Sourced from a small sample set, sampled samples< t j , l j The data originates from the sampled sample set. The sample data used in text concatenation and enhancement can also all originate from the sampled sample set, i.e., the data samples... t i , l i >and sampled samples< t j , l j All samples are derived from the sampled dataset.

[0098] After constructing the text concatenation sample set and the hybrid augmentation sample set, the intelligent device can input the obtained text concatenation sample set and the hybrid augmentation sample set together with the original text to be classified into the training model to complete the model training. For example, if the original data of the text to be classified is D0, the text concatenation sample set obtained by text concatenation is D1, and the hybrid augmentation sample set obtained by hybrid augmentation is D2, then during model training, the original data D0, the text concatenation sample set D1, and the hybrid augmentation sample set D2 can be merged and input into the training model to obtain the classification results corresponding to each training data.

[0099] Since the training data after data merging includes the original data D0, as well as the text splicing sample set D1 and the hybrid augmentation sample set D2 supplemented by data augmentation, the amount of data in a few sample sets in the original data will be expanded. That is, in some embodiments, the merged result of performing data merging on the text to be classified, the text splicing sample set, and the hybrid augmentation sample set is called balanced training data.

[0100] After expanding the training data, the amount of data corresponding to each classification label is:

[0101]

[0102]

[0103]

[0104] In hybrid augmentation, smart devices incorporate embedding vectors into the text representation using a soft approach. This represents the amount of data that has been partially incorporated, and the data amount obtained through this calculation method may be a floating-point number. Therefore, since the augmented training data is class-balanced, we can assume that the final amount of data for each class is the same, and thus:

[0105]

[0106] Obviously, One of its analytical solutions is given by the following data size for each label:

[0107]

[0108] For ease of description, we can let , .but This indicates the proportion of data generated by the text merging method. This indicates the proportion of data generated by the hybrid enhancement method. This indicates the proportion of data that was partially incorporated.

[0109] After the balanced training data obtained by merging is input into the training model, the model can output a classification result for the input data, that is, the classification probability of each input data relative to the class category. The intelligent device then compares the classification probability output by the model with the classification labels of the training samples to determine the error between the classification probability and the classification label. Based on the determined error, the model parameters of the training model are adjusted by backpropagation. This allows the training model to go through the "input-output-adjustment" process corresponding to multiple training samples, gradually improving, and finally obtaining a training model that can output accurate classification results.

[0110] To determine the error between the classification probability and the classification label, smart devices can invoke a loss function during model training. A loss function, also known as a cost function, maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event. In applications, the loss function can be used as a learning criterion related to optimization problems; that is, the model is solved and evaluated by minimizing the loss function.

[0111] The loss function used during model training can be based on a specific loss calculation method. For example, the cross-entropy loss function can be used for loss calculation during model training. The cross-entropy loss function is a smoothing function, essentially an application of cross entropy from information theory to classification problems. As defined, minimizing cross entropy is equivalent to minimizing the relative entropy between the observed and estimated values, i.e., the Kullback-Leibler divergence of their probability distributions. Therefore, the cross-entropy loss function can provide an unbiased surrogate loss estimate.

[0112] In this embodiment, the balanced training data input to the training model consists of three types of training data: the original training text data (the text to be classified), the text concatenation sample set, and the mixed augmentation sample set. Different data can have different semantic contributions during the actual classification process. Different loss functions can be used to calculate the loss for different parts of the data. That is, the training model uses different loss functions to calculate the loss for the text to be classified, the text concatenation sample set, and the mixed augmentation sample set, so as to update the model parameters of the training model through backpropagation.

[0113] like Figure 10 As shown, in some embodiments, in order to update the model parameters of the training model, the smart device can, after inputting the text to be classified, the text concatenation sample set, and the hybrid augmented sample set into the training model, first set the loss function of the training model according to the composition of the input data. The loss function includes a first loss function and a second loss function. The first loss function is used to calculate the loss of the classification results corresponding to the text to be classified and the text concatenation sample set; the second loss function is used to calculate the loss of the classification results of the hybrid augmented sample set.

[0114] For example, for the original training text data D 0 represents the initial dataset, which is unenhanced and therefore has the highest data quality. However, for the text concatenation sample set... D 1. Even with text concatenation and fusion, the correct labels after concatenation are still represented as 0-1. This shows that for the original data... D 0 and text concatenation sample set D 1. FocalLoss, based on cross-entropy, can be used as the loss function for training. Specifically, the FocalLoss loss function is as follows:

[0115]

[0116] In the formula, This is the loss value; γ represents the classification probability output by the trained model, and γ is a preset hyperparameter.

[0117] In multi-label scenarios, since a single data point can be labeled with multiple classification labels, the loss can be calculated by averaging the results after calculating the loss for each category. That is, for training data with n categories, the loss can be calculated using the following formula:

[0118]

[0119] In the formula, L For the total loss, For category i The corresponding loss value, n This represents the number of categories.

[0120] However, for the mixed-augmented sample set D2, since both text and labels undergo mixup enhancement, using cross-entropy as the loss function will cause problems. Therefore, mean squared error (MSE) can be used as its loss function. That is, for training data with n classes, the loss can be calculated using the following formula:

[0121]

[0122] In the formula, L For the total loss, n For the number of categories, y i The category label values ​​obtained after mixing in mixup. The classification probability output by the training model.

[0123] Based on the above loss function calculation method, the intelligent device can sequentially calculate the loss corresponding to each data set, and then use the loss function to calculate the final loss, so as to backpropagate the final loss and adjust the model parameters of the trained model. The final loss is a weighted sum of the losses corresponding to the text to be classified, the text concatenation sample set, and the mixed augmented sample set.

[0124] In some embodiments, when calculating the final loss using a loss function, the smart device can first input balanced training data into the training model, then obtain the classification results output by the training model, and calculate the losses corresponding to the text to be classified and the text concatenation sample set according to the first loss function and the classification results respectively; and calculate the loss corresponding to the hybrid augmented sample set according to the second loss function and the classification results. Finally, the losses corresponding to the text to be classified, the text concatenation sample set, and the hybrid augmented sample set are weighted and summed according to preset initial weights.

[0125] For example, to calculate the final loss, different weights can be pre-assigned to the three datasets as ρ0, ρ1, and ρ2. Based on the loss function described above, the original data can be weighted accordingly. D 0. Calculate the loss L 0, for the text splicing sample set D 1. Calculate the loss L 1, D 2. Calculate the loss L 2. The final loss can then be expressed as:

[0126]

[0127] In the formula, For the final loss, ρ0, ρ1, and ρ2 are preset hyperparameters used to represent the weights. For example, the initial value of ρ0 can be set to 1, and the initial value of ρ1 can be set to... That is, the product of the first efficiency parameter θ and the average classification probability. The initial value of ρ2 can be set to That is, the product of the second efficiency parameter γ and the average classification probability, plus the sum of the proportion of data partially incorporated.

[0128] As can be seen from the above technical solutions, intelligent devices can determine the final loss using the aforementioned loss function, and then backpropagate the final loss to update the model parameters of the trained model. Therefore, the text classification method provided in the above embodiments can use different loss functions in the backpropagation stage of model training to obtain the final loss. Thus, while increasing the amount of data in a small sample set, it enables the training model to be trained simultaneously on text concatenation sample sets, hybrid augmented sample sets, and the original labeled data, thereby improving the robustness of the final trained model.

[0129] Based on the aforementioned imbalanced multi-label text classification method, some embodiments of this application also provide a smart device, including: a storage module and a processing module. The storage module is configured to store a training model; as... Figure 5 , Figure 11 As shown, the processing module is configured to execute the following program steps:

[0130] Obtain the text to be classified, which is training sample data with multiple classification labels;

[0131] Sampling is performed on the text to be classified to obtain a sample set;

[0132] Construct a text splicing sample set, which is formed by splicing sample data from the sampling sample set;

[0133] Construct a hybrid augmented sample set, which is formed by mixing the numerical vectors corresponding to the sample data in the sampled sample set. The numerical vectors are formed by the transformation of the text to be classified through the network layer in the training model.

[0134] The text to be classified, the text concatenation sample set, and the hybrid augmentation sample set are input into the training model. The training model uses different loss functions to calculate the loss for the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set, so as to update the model parameters of the training model through backpropagation.

[0135] As can be seen from the above technical solutions, the method provided in this application, including the intelligent device, can obtain a sample set by sampling the text to be classified after acquiring the text to be classified. A text concatenation sample set and a hybrid augmentation sample set are then constructed. Finally, the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set are input into the training model for model training. This method can construct the text concatenation sample set and the hybrid augmentation sample set by concatenating sample data and mixing the corresponding numerical vectors, thus expanding the amount of sample data and alleviating the problem of low accuracy caused by imbalanced sample data. Furthermore, different loss functions are used to calculate the loss for different datasets to update the model parameters of the training model through backpropagation, thereby improving the accuracy of model calculation.

[0136] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. An imbalanced multi-label text classification method, characterized in that, include: Obtain the text to be classified, which is training sample data with multiple classification labels, including a majority sample set and a minority sample set; Sampling is performed on the text to be classified to obtain a sample set; Construct a text splicing sample set, which is formed by splicing sample data from the minority sample set and the sampled sample set; Input the training data from the text to be classified into the network layer of the training model; Obtain the digital vector output by the network layer, wherein the digital vector is generated by the network layer performing vector mapping on the training data; Set a mixing ratio parameter, which follows a uniform distribution or a Beta distribution; Based on the mixing ratio parameter, the mixup algorithm is executed on the numerical vectors corresponding to the minority sample set and the sampled sample set to generate a mixed enhanced sample set; Data merging is performed on the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set to obtain balanced training data; The training model is configured with a loss function, which includes a first loss function and a second loss function. The first loss function is used to calculate the loss of the classification result corresponding to the text to be classified and the text concatenation sample set. The second loss function is used to calculate the loss of the classification result of the hybrid augmented sample set. The final loss is calculated using the loss function, and the final loss is the weighted sum of the losses corresponding to the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set. The final loss is backpropagated to adjust the model parameters of the trained model.

2. The imbalanced multi-label text classification method according to claim 1, characterized in that, After obtaining the text to be classified, the method further includes: Iterate through the number of samples of the text to be classified and the number of times each classification label appears in the text to be classified; Calculate the probability of occurrence for each category label, where the probability of occurrence is the ratio of the number of occurrences corresponding to the category label to the number of samples; The sample data in the text to be classified is divided according to the occurrence probability, and the sample data includes a majority sample set and a minority sample set.

3. The imbalanced multi-label text classification method according to claim 2, characterized in that, The steps of classifying sample data in the text to be classified according to the occurrence probability include: The occurrence probability is compared with the sample judgment threshold, where the sample judgment threshold is the average occurrence probability of all the classification labels. If the probability of the current classification label is greater than or equal to the sample judgment threshold, the training sample data corresponding to the current classification label is marked as belonging to the majority sample set; If the occurrence of the current classification label is less than the sample judgment threshold, the training sample data corresponding to the current classification label is marked as belonging to the minority sample set.

4. The imbalanced multi-label text classification method according to claim 2, characterized in that, The steps for performing sampling in the text to be classified include: Obtain a first efficiency parameter and a second efficiency parameter. The first efficiency parameter is used to calculate the number of samples in the sample set of the spliced ​​sample set. The second efficiency parameter is used to calculate the number of samples in the sample set corresponding to the hybrid augmented sample set. Calculate the sampling median value, which is the ratio of the average occurrence probability of all the category labels to the occurrence probability of the current category label; A first number of sample data is sampled from the text to be classified to generate a first sample set, wherein the first number is equal to the product of the first efficiency parameter and the sampling median value; A second number of sample data is sampled from the text to be classified to generate a second sample set, the second number being equal to the product of the second efficiency parameter and the sampling median value.

5. The imbalanced multi-label text classification method according to claim 4, characterized in that, The method further includes: In the step of constructing a text concatenation sample set, concatenation is performed on at least two sample data from the minority sample set and the first sampled sample set to construct the text concatenation sample set; In the step of constructing the hybrid augmented sample set, the numeric vectors corresponding to at least two sample data in the minority sample set and the second sampled sample set are mixed to generate the hybrid augmented sample set.

6. The imbalanced multi-label text classification method according to claim 1, characterized in that, The steps to construct a text concatenation sample set include: Call the concatenation function, which is used to concatenate at least two sample data; The concatenation function is used to perform text concatenation on the minority sample set and the sampled sample set to obtain concatenated text, which includes delimiters; Each of the concatenated texts is assigned a concatenation label in turn to generate a concatenated sample set, wherein the concatenation label is the union of the classification labels corresponding to the concatenated sample data.

7. The imbalanced multi-label text classification method according to claim 1, characterized in that, The steps for calculating the final loss using the loss function include: Input the balanced training data into the training model; Obtain the classification result output by the trained model; Based on the first loss function and the classification result, calculate the loss corresponding to the text to be classified and the text concatenation sample set, respectively. Calculate the loss corresponding to the hybrid augmented sample set according to the second loss function and the classification result; Based on preset initial weights, the corresponding losses of the text to be classified, the text splicing sample set, and the hybrid augmented sample set are weighted and summed.

8. A smart device, characterized in that, include: The storage module is configured to store the training model; The processing module is configured as follows: Obtain the text to be classified, which is training sample data with multiple classification labels, including a majority sample set and a minority sample set; Sampling is performed on the text to be classified to obtain a sample set; Construct a text splicing sample set, which is formed by splicing sample data from the minority sample set and the sampled sample set; The training data from the text to be classified is input into the network layer of the training model; Obtain the digital vector output by the network layer, wherein the digital vector is generated by the network layer performing vector mapping on the training data; Set a mixing ratio parameter, which follows a uniform distribution or a Beta distribution; Based on the mixing ratio parameter, the mixup algorithm is executed on the numerical vectors corresponding to the minority sample set and the sampled sample set to generate a mixed enhanced sample set; Data merging is performed on the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set to obtain balanced training data; The training model is configured with a loss function, which includes a first loss function and a second loss function. The first loss function is used to calculate the loss of the classification result corresponding to the text to be classified and the text concatenation sample set. The second loss function is used to calculate the loss of the classification result of the hybrid augmented sample set. The final loss is calculated using the loss function, and the final loss is the weighted sum of the losses corresponding to the text to be classified, the text concatenation sample set, and the hybrid augmentation sample set. The final loss is backpropagated to adjust the model parameters of the trained model.

Images