Text multi-classification method and system based on weighted contrast learning and application

Abstract

The invention discloses a text multi-classification method based on weighted contrast learning. The text multi-classification method comprises the following steps: constructing a positive and negative sample pair data set Dpair by using a labeled data set D1; a model phi 1 used for conducting similar score prediction on the two input texts is trained on the Dpair; training a text encoder phi 2 by using the D1, calculating a data representation vector of the D1 by using the text encoder phi 2, and clustering the representation vector in a high-dimensional hidden layer space to obtain a set C of clustering centers; calculating a characterization vector of the Du by using phi 2, and calculating the distance from each data characterization vector to each clustering center in the C; delimiting a distance gamma as a target distance, filtering the data, the distance of which to the nearest clustering center is greater than gamma, in all Du, and retaining label-free data, the distance of which is less than gamma, so as to obtain a filtered label-free data set Dufilled; a text classifier [phi] 3 is trained on the D1 and the Dufilled so as to effectively classify the texts; and calculating the probability that the sample belongs to each category and the category thereof by using a text classifier phi 3. The method has good effectiveness, robustness and label utilization rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of text classification, and in particular relates to a text multi-classification method, system and application based on weighted contrastive learning and data filtering when there is less labeled data and more unlabeled data. Background Art

[0002] Text classification is a fundamental task in natural language processing, widely used in practical scenarios such as sentiment analysis, topic tagging, and recommender systems. When processing large-scale, dynamic data such as social media, traditional manual tagging methods are time-consuming and subject to subjective factors. Therefore, automated and reliable text classification models have attracted much attention. Text classification methods are mainly divided into traditional methods and deep learning methods. Traditional methods use feature extractors and classifiers, such as Naive Bayes and Support Vector Machines. Deep learning models such as TextCNN and BERT have made significant progress in recent years, leveraging convolutional neural networks and bidirectional encoding techniques to improve text classification performance. Deep learning models are gradually replacing traditional methods and becoming the automatic text classification solution sought by researchers.

[0003] However, the methods proposed above are often used when there is sufficient labeled data. When only limited labeled data is available, complex neural networks are prone to overfitting, and the models often fail to capture the characteristics of different text categories. Furthermore, in real-world scenarios, a large number of unlabeled samples are often available. Therefore, a growing number of text classification studies are turning to semi-supervised learning (SSL) to effectively utilize large amounts of unlabeled data to address these issues and help models better learn text representations. Contrastive learning is a currently popular type of self-supervised learning method. Its basic idea is to learn the structure and semantic information of the feature space by maximizing the differences between samples. This method typically compares a group of similar samples with a group of dissimilar samples, enabling the model to cluster similar samples together and separate dissimilar samples. The core of contrastive learning is to learn a representation learning network that maps input samples into a low-dimensional feature space, where similar samples are close together and dissimilar samples are far apart.

[0004] However, existing work using contrastive learning for text representation often relies on the consistency assumption, assuming that samples after data augmentation still belong to the same category as the original data. However, when data augmentation is applied to text, the semantics of the augmented samples may fluctuate. Whether a sample's semantics change after data augmentation is uncontrollable. Indiscriminately assuming that samples before and after data augmentation belong to the same category can create inconsistencies in model training and affect classification performance.

[0005] In addition, the model performs poorly in the early stages of training, as the amount of unlabeled data is large and complex, and there are some samples that are difficult to distinguish. Figure 1 As shown in the figure, simple samples are far from the classification hyperplane and are unlikely to change to other classes after data augmentation for contrastive learning. However, when using difficult samples for contrastive learning, since the samples themselves are at the intersection of the classification hyperspheres, the two enhanced samples obtained after data augmentation have a higher probability of changing to different classes. In other words, these samples are more likely to have semantic inconsistencies during data augmentation. Such difficult samples will interfere with the model and hinder its learning. Summary of the Invention

[0006] In order to address the shortcomings of the existing technology, the purpose of the present invention is to provide a semi-supervised text multi-classification method when there is less labeled data and more unlabeled data. This method has higher Macro F1-score and Micro F1-score than the existing methods when performing classification.

[0007] The present invention is based on the semi-supervised method of contrastive learning. The core of contrastive learning is to learn a representation learning network that maps input samples to a low-dimensional feature space so that similar samples are closer in this space, while dissimilar samples are farther apart. Existing contrastive learning methods suffer from the problem of label floating when performing data augmentation. That is, the semantics of the two enhanced samples obtained after one sample data is augmented undergo drastic changes, which will mislead the model learning. Unlike existing contrastive learning methods, the present invention proposes a weighted contrastive learning method to solve this problem. The weighted contrastive learning refers to designing a contrastive learning guidance module based on a twin network to predict the similarity score of the two input texts, and using the predicted similarity score as the weight to weight the original contrastive learning information loss function. In addition, due to the large amount and complex types of unlabeled data, there will be some samples with relatively ambiguous semantics and difficult categories to distinguish. Compared with existing semi-supervised text classification algorithms, the present invention proposes an operation to filter the samples that are more difficult to classify in the unlabeled data to make the model more efficient when learning unlabeled data.

[0008] The specific technical solution for achieving the purpose of the present invention is: a text multi-classification method based on weighted contrastive learning, including five steps: constructing a positive and negative sample pair dataset, training a text similarity discrimination model, filtering difficult samples in unlabeled data, training a text classifier using weighted contrastive learning, and calculating the probability that the sample belongs to each category. The method includes the following specific steps:

[0009] Step 1: Use a labeled dataset D l Construct a positive and negative sample pair dataset Dpair , the labeled dataset D l The uniqueness of the present invention is that K labeled sample data can be used to construct K 2 Positive and negative sample pairs are used for the subsequent training of the text similarity discrimination model. The construction method is to randomly select other samples of the same category to form positive sample pairs after selecting a sample, and randomly select samples of other categories to form negative sample pairs. This combination method can make the number of positive and negative sample pairs equal to the square of the amount of labeled data. The positive and negative sample pairs include positive sample pairs and negative sample pairs, each sample pair includes two text data and a label corresponding to the sample pair; the positive sample pair label is 1, indicating that the two text data are similar; the negative sample pair label is 0, indicating that the two texts are not similar; the specific steps of step 1 to generate the positive and negative sample pair data set are as follows:

[0010] Step 1-1: There will be a labeled dataset D l Classify them according to their respective categories and obtain the data sets of each category. The data sets of each category are expressed as follows: D c ={D1, D2, ..., D N}; Step 1.1 Classify the sample data of each category according to the label to form a set of each category;

[0011] Step 1-2: Construct positive sample pairs. A positive sample pair refers to two samples that are semantically similar and belong to the same category. The purpose of constructing positive sample pairs is to enable the text similarity discrimination model to learn the common points between the same samples. The specific construction steps are as follows:

[0012] Step 1-2-1: Traverse all sample data in the labeled dataset, and set the currently traversed sample as x i ;

[0013] Step 1-2-2: Determine D c Zhong and x i The sample set D of the same category k , in D k Randomly sample a sample x from pos , we need to ensure x i with x pos Different and combined (x i , x pos ) and (x pos , x i ) has never been selected before. In this process, the parameter N can be set to represent the sample x i The number of positive sample pairs is constructed. After setting this parameter, just repeat this step N times to obtain the number of positive sample pairs x. iConstruct N positive sample pairs. In addition, in this process, the sample combination may be exhausted. In order to avoid being unable to perform subsequent operations after being exhausted, it is set that a sample pair random sampling process is performed at most θ times. If after θ times, no sample that satisfies x is found, i and x pos If the sample does not meet the conditions, the sampling of the sample is terminated and the process returns to step 1-2-1 to continue traversing the labeled data set.

[0014] The maximum number of random sampling times θ and the sample set D k The number of samples in the sample set D k When the number of samples is small, it is easy to sample the same sample as the previous one or several times, so a larger sampling number θ is selected; otherwise, a smaller sampling number θ is selected;

[0015] Step 1-2-3: Set the label of the positive sample pair obtained by screening to 1, which is a positive sample pair;

[0016] Step 1-2-4: Repeat steps 1-2-1 to 1-2-3 until all samples in the labeled dataset are traversed.

[0017] Steps 1-3: Constructing negative sample pairs. Negative sample pairs refer to two samples from different classes that are semantically distinct. Constructing negative sample pairs is primarily to enable the text similarity discrimination model to learn the distinction between classes. This also prevents the model from collapsing during training, where all inputs are mapped to a fixed point or a small area due to learning only from positive sample pairs. The specific steps for constructing negative sample pairs are as follows:

[0018] Step 1-3-1: Traverse all sample data in the labeled dataset, and set the currently traversed sample as x j ;

[0019] Step 1-3-2: Determine D c Zhong and x j The sample set D of the same category k , after removing D k Data set D c / D k Randomly sample a sample x from neg , we need to ensure x neg with x j Different and combined (x j , x neg ) and (x neg , x j ) has never been selected before. Similar to the way of constructing positive sample pairs, in this process, the parameter M can be set to represent the sample x jThe number of negative sample pairs constructed. After setting this parameter, just repeat this step M times to obtain the number of negative sample pairs. j Construct M negative sample pairs. Similarly, to avoid exhausting the sample combinations, set the random sampling process of a sample pair to be performed at most θ times. If no sample that meets the above conditions is found after θ times, the sampling of the sample is terminated and return to step 1-3-1 to continue traversing the labeled dataset;

[0020] Step 1-3-3: Set the label of the negative sample pair obtained by screening to 0, that is, the negative sample pair;

[0021] Step 1-3-4: Repeat steps 1-3-1 to 1-3-3 until all samples in the labeled dataset are traversed.

[0022] Step 1-4: Combine the positive sample pairs obtained in step 1-2 and the negative sample pairs obtained in step 1-3 to obtain the positive and negative sample pair dataset D pair .

[0023] Step 2: In the positive and negative sample dataset D pair Train a text similarity discrimination model Φ1( Figure 2), which is used to predict the similarity scores of two input texts; the text similarity discrimination model consists of a text encoder and a multi-layer neural network. The text encoder mainly converts text data into corresponding feature vectors, and the multi-layer neural network is used to predict the spliced text vectors to obtain the similarity scores of the two input texts. Unlike previous text classification based on contrastive learning, the uniqueness of the present invention lies in the design of a text similarity discrimination model Φ1 to guide the contrastive learning process. The conventional process of contrastive learning is to first perform data enhancement on samples in a batch. After that, contrastive learning is based on the consistency assumption, that is, "two samples obtained by data enhancement of the same sample are similar to each other, and samples obtained by data enhancement of different samples are different from each other", and the infonce loss function is used for model training, with the aim of making the representations of similar samples closer and the representations of dissimilar samples farther apart. However, the consistency assumption in the above process is relatively ideal. In actual situations, it may happen that the two samples obtained after data enhancement of the same sample are not similar to each other. The present invention uses a similarity discrimination model to calculate the similarity of the samples after data enhancement, and adds the calculated similarity score as a weight into the Infonce loss function to guide the contrastive learning process, thereby alleviating the problem of label floating in the contrastive learning process, improving the contrastive learning effect, and further improving the text classification effect. The text similarity discrimination model Φ1 consists of a text encoder E1 based on a BERT pre-training model and a multi-layer neural network F that outputs a similarity score. The text encoder E1 is used to map the original text data into a high-dimensional vector, while the multi-layer neural network F maps the high-dimensional vector into a numerical scalar. The entire text similarity discrimination model Φ1 is trained by performing cross-entropy loss on the numerical scalar calculated from the input text pair and its label; after the model converges, the similarity degree of the input sample pair can be predicted. The specific steps for constructing a text similarity discrimination model are as follows:

[0024] Step 2-1: The positive and negative samples generated in step 1 are combined into the dataset D pair Input the original text similarity discrimination model Φ1;

[0025] Step 2-2: Calculate the high-dimensional representation vector of the sample pair. The text encoder E1 will map the input sample pair text1 and text2 into two text representation vectors v1 and v2 of the same dimension.

[0026] Step 2-3: Vector concatenation: Concatenate the two text representation vectors v1 and v2 obtained in step 2-2 to obtain the concatenated high-dimensional representation vector v3 of the sample pair.

[0027] Step 2-4: Similarity score prediction. The concatenated vector v3 obtained in Step 2-3 is input into the multi-layer neural network F. The network maps the high-dimensional vector v3 into a scalar value s, which serves as the prediction of the similarity score of the two input texts.

[0028] Step 2-5: Minimize the cross entropy loss function so that the similarity score s predicted by the multi-layer neural network is close to the label of the input sample pair.

[0029] The formula of the cross entropy loss function is as follows:

[0030] loss=-(ylog(s)+(1-y)log(1-s))

[0031] Where s is the similarity score predicted by the model, and y is the label of the text pair. If the input is a positive sample pair, y is 1, and if it is a negative sample pair, y is 0. The optimization goal of the text similarity discrimination model is to minimize the cross entropy loss function.

[0032] Step 2-6: Repeat steps 2-1 to 2-5 until the cross entropy loss function of the text similarity discrimination model Φ1 no longer decreases, the model converges, and the optimized text similarity discrimination model is obtained.

[0033] Step 3: For the unlabeled dataset D u Filter the difficult samples in the dataset to obtain the filtered unlabeled dataset D u _filtered Different from the previous text classification based on contrastive learning, the uniqueness of this invention is that a data filtering module is designed to classify the unlabeled dataset D u Perform difficult sample filtering before contrastive learning. Figure 1 As shown, the simple sample refers to the distance classification hyperplane ( Figure 1 The multiple circles in the image are relatively far away, and it is not easy to become a sample of another category in the high-dimensional representation space after the data enhancement operation of contrastive learning; difficult samples refer to samples that are at the intersection of the hyperspheres of each classification during contrastive learning. The two enhanced samples obtained after data enhancement have a greater probability of becoming samples of different categories. Difficult samples are more likely to have semantic inconsistency problems during data enhancement, which will cause certain interference to model learning when the model performance is poor in the early stage of model training. Therefore, the present invention designs a difficult sample filtering module, and the specific filtering steps are as follows:

[0034] Step 3-1: Use labeled dataset D l A text encoder Φ2 is obtained by training. The specific network structure is as follows Figure 3As shown in the figure, the entire model consists of a text encoder Φ2 (i.e., the encoder in the figure) and a multi-layer neural network G for classification. The model is trained on the network using the classification task using labeled sample data until the model converges, and a text encoder Φ2 with good text feature extraction capabilities is obtained.

[0035] The text encoder Φ2 is based on the BERT pre-training model, maps the input original text into a high-dimensional vector, then inputs the obtained high-dimensional vector into a multi-layer neural network G, and outputs a one-dimensional vector. Each value in the vector represents the predicted probability that the text belongs to that category. Then, a multi-classification cross entropy loss function is used to match the predicted category of the input text with its label. The optimization is continuously carried out until the multi-classification cross entropy loss function no longer decreases and the model converges.

[0036] The specific steps to obtain the text encoder Φ2 are as follows:

[0037] Step 3-1-1: Calculate text representation. The text encoder Φ2 will map the input text sample text into a high-dimensional text representation vector v;

[0038] Step 3-1-2: Label prediction. Input the text representation vector v obtained in step 3-1-1 into the multi-layer neural network G. This network maps the high-dimensional vector v into a vector logits. The dimension of the vector logits is 1×R, where R is the number of label categories. Each value in the logits represents the probability that the sample belongs to the corresponding category.

[0039] Step 3-1-3: Use the multi-classification cross entropy loss function to make the predicted logits close to the label of the input sample. The formula of the loss function is as follows:

[0040]

[0041]

[0042] Where B is the batch size. is the probability that the model predicts that sample i belongs to the kth class, is the label of sample i in the kth category. If sample i belongs to the kth category, then is 1 if the value is set, otherwise it is 0. is the probability predicted by the softmax function of logits. The goal of the text encoder model is to minimize the multi-classification cross entropy loss function.

[0043] Step 3-1-4: Repeat steps 3-1-1 to 3-1-3 until the model's multi-classification cross entropy loss function no longer decreases and the model converges. The text encoder Φ2 will be used to extract features from unlabeled data later.

[0044] Step 3-2: Find the representation of all labeled sample data. l All sample data in the text encoder Φ2 in step 3-1 are input to obtain the text representation vector V of the labeled sample data l ;

[0045] Step 3-3: The text representation vector V obtained in step 3-2 l Use the k-means algorithm to perform clustering operations and obtain the cluster centers {C1, C2, ..., C class_cnt}, where C class_cnt is the number of categories; the specific steps of k-means are as follows:

[0046] Step 3-3-1: Randomly select V l Middle C class_cnt data points as the initial cluster centers;

[0047] Step 3-3-2: For each data point, calculate its distance to the center of each cluster and assign the data point to the cluster with the closest distance;

[0048] Step 3-3-3: For each cluster, calculate the mean of all its member data points and use the mean as the new cluster center;

[0049] Step 3-3-4: Repeat steps 3-4-2 and 3-4-3 until the change in cluster center is small or the predetermined number of iterations is reached;

[0050] Step 3-3-5: When the cluster center no longer changes significantly or reaches the predetermined maximum number of iterations, the algorithm converges and the cluster centers {C1, C2, ..., C class_cnt}.

[0051] Step 3-4: Filter the unlabeled data. Define a target distance γ and filter out all the data whose distance from the nearest cluster center exceeds γ. The specific steps are as follows:

[0052] Step 3-4-1: Traverse the unlabeled dataset D u , let the current unlabeled data be x;

[0053] Step 3-4-2: Use the text encoder Φ2 obtained in step 3-2 to encode x to obtain its representation vector v;

[0054] Step 3-4-3: Calculate v to {C1, C2, ..., C class_cnt The distance from the cluster center is calculated, and the distance to the nearest cluster center is recorded as d. At the same time, the category to which the nearest cluster center belongs is the category to which the current unlabeled data belongs;

[0055] Step 3-4-4: Set a target distance γ. If d≤γ, retain the unlabeled data x, otherwise filter out x. Arrange the unlabeled data assigned to each category from small to large according to the distance from these unlabeled data to the cluster center of their category. The distance from the unlabeled data at the median position to the cluster center is the value of γ.

[0056] Step 3-5: Repeat steps 3-4-1 to 3-4-4 until all unlabeled data have been filtered, and the filtered unlabeled dataset D is obtained. u-filtered .

[0057] Step 4: In the labeled dataset D l And the filtered unlabeled dataset D u_filtered A text classifier Φ3 is trained on . The network structure is as follows Figure 4 As shown, the training of the text classifier Φ3 proposed in this invention includes two parts: using the labeled dataset D l Perform supervised training of the model and use the filtered unlabeled dataset D u_filtered Perform unsupervised training. The supervised training is done on a labeled dataset D l Use L cls The loss function makes the model predict the category of the text close to its label. The unsupervised training is performed on the filtered unlabeled dataset D u_filtered Use L contrastive (Weighted infonce) makes the samples obtained from data augmentation that are mutually positive examples close to each other in the representation space, and samples that are mutually negative examples far away from each other in the representation space. The weighted infonce loss function is calculated by multiplying the infonce loss function by the similarity weight between the two input samples. This weight is represented by the similarity score between the two input samples predicted by the text similarity discrimination model Φ1 obtained in step 2. The specific steps of training the text classifier Φ3 are as follows:

[0058] Step 4-1: Initialize a text encoder E2 using the BERT pre-training model and randomly initialize a multi-layer neural network H;

[0059] Step 4-2: There will be a labeled dataset D lInput the text encoder E2 to obtain a high-dimensional representation vector of the text, then input the high-dimensional representation vector into the multi-layer neural network H, and output a one-dimensional vector. Each value in the vector represents the predicted probability that the text belongs to that category;

[0060] Step 4-3: Using L cls The loss function makes the predicted probability obtained in step 4-2 close to the label of the text;

[0061] Specific L cls The formula of the loss function is as follows:

[0062]

[0063] Among them, Q is the number of categories; P is the size of the batch; y ic is a sign function (valued as 0 or 1), if the true label of sample i is c, it takes 1, otherwise it takes 0; p ic is the probability that sample i belongs to category c;

[0064] Step 4-4: Use the filtered unlabeled dataset D u_filtered Perform contrastive learning. The contrastive learning first converts the unlabeled data sample x u Perform two data enhancements to obtain two enhanced samples a i and a j ; Then we will enhance sample a i and a j Input the text encoder E2 respectively to get the enhanced sample a i and a j Characterization vector z i and z j ;

[0065] Step 4-5: Using L contrastive The loss function makes the enhanced sample representation vector z obtained in step 4-4 i and z j Proximity is achieved in representational space;

[0066] Specific L contrastive The formula of the loss function is as follows:

[0067]

[0068]

[0069] Among them 1 [k≠i]∈{0, 1} is the indicator function, and τ is the temperature coefficient, which can help the model perform better on difficult negative samples. If the temperature coefficient is set larger, the distribution of the loss function in step 7 becomes smoother, and the contrast loss will treat all negative samples equally, resulting in a lack of emphasis on model learning. If the temperature coefficient is set too small, the model will pay more attention to particularly difficult negative samples, but at the same time, those negative samples are likely to be potential positive samples, which will make it difficult for the model to converge or have poor generalization ability. Therefore, the role of the temperature coefficient is to control the model's discrimination against negative samples. i is the i-th sample, z j is the sample generated by data enhancement of the i-th sample, z k It represents all samples in the same batch and their data-enhanced sample sets except z i Samples selected from outside. sg (z i , z j ) is z i and z j The similarity score output after inputting the text similarity discrimination model Φ1;

[0070] Step 4-6: Repeat steps 4-2 to 4-5 until the L of the text classifier Φ3 is contrastive The loss function no longer decreases and the model converges. The specific formula of the overall objective function L is as follows:

[0071] L=L cls +α·L contrastive

[0072] Where α is a hyperparameter used to adjust L cls and L contrastive The ratio between them.

[0073] Step 5: Use the text classifier Φ3 to calculate the probability that the sample belongs to each category and determine the category to which the sample belongs. After the model converges, the following calculation method can be used to obtain the category to which the model predicts the sample belongs:

[0074] logits=Φ3(x)

[0075]

[0076]

[0077] l pred =argmax(softmax(logits))

[0078] Where Φ3 is the text classifier trained in step 4, x is the input text, logits represents the probability that the text belongs to each category after the text is input into the text classifier, softmax(·) is used to calculate the proportion of each value in the input vector, argmax(·) function is used to obtain the subscript index corresponding to the maximum value in the input vector, l pred It is the category to which the input text belongs predicted by the final text classification model Φ3.

[0079] The present invention provides a multi-classification system for implementing the above-mentioned text multi-classification method, wherein the multi-classification system comprises: a positive and negative sample pair construction module, a text similarity discrimination module, an unlabeled difficult sample filtering module, and a semi-supervised text multi-classification module;

[0080] The positive and negative sample pair construction module is used to generate a positive and negative sample pair data set using labeled sample data, and use the positive and negative sample pair data set to train a text similarity discrimination model;

[0081] The text similarity discrimination module is used to guide the subsequent unlabeled data to perform contrastive learning. Taking into account the problem of sample label floating when using contrastive learning to perform data enhancement on text, that is, the two enhanced samples obtained after the same sample data is enhanced may not necessarily still belong to the same category semantically, and different samples may be semantically similar, for example, synonyms, synonymous sentences, etc. may appear in the text. If it is based on the consistency assumption without judgment (that is, the two samples obtained by enhancing the same sample data must belong to the same category, and different samples must belong to different categories), the model is bound to produce contradictions during the training process, thereby affecting the model effect. Therefore, the present invention proposes a text similarity discrimination module based on a twin network to solve the above problem. Specifically, the input of the text similarity discrimination module is two samples, and it can output the similarity score between the two samples. The similarity score is used as the weight coefficient in the contrastive learning loss function to guide the contrastive learning process, that is, whether to pull the two samples after data enhancement closer or push them away.

[0082] The unlabeled difficult sample filtering module is used to mitigate the interference of difficult samples (samples that are difficult to classify) in the early stages of model training. Model performance is poor in the early stages of training because the amount of unlabeled data is large and complex, and some samples are difficult to distinguish. These difficult samples can interfere with the model and hinder its learning. To address this issue, this paper designs a clustering-based data filtering module to screen unlabeled data and filter out difficult samples.

[0083] The semi-supervised text multi-classification module is used to integrate the text similarity discrimination module and the unlabeled difficult sample filtering module, and uses supervised tasks and unsupervised tasks to train the model simultaneously, so that the model can efficiently utilize labeled data while also obtaining information about the encoded text from unlabeled data, thereby realizing multi-classification of text.

[0084] The present invention also provides the application of the above-mentioned text multi-classification algorithm in news text classification and social media dynamic text classification. For example, the current popular social media information platform will generate tens of millions of dynamics every day. In order to be able to analyze these dynamics, product operations need to classify the categories of dynamics. However, there is too little labeled data. Faced with a large amount of dynamic data generated, manual labeling is time-consuming and labor-intensive, and is far from the speed of data generation. In addition, manual labeling will be affected by human factors. A semi-supervised text multi-classification model that can efficiently utilize a small amount of labeled data and a large amount of unlabeled data becomes crucial. The present invention is applicable to the above scenarios, and is also applicable to the classification of news text data and the classification of data composed of various types of text, with a high F1-score.

[0085] Compared with the existing technology, the present invention provides a new and universal method for the actual needs of real text multi-classification scenarios. It has good applicability and can be directly applied to various types of tasks. The method is simple and efficient, and has the following beneficial technical effects:

[0086] (1) Compared with existing contrastive learning methods, the present invention provides a contrastive learning guidance module, which alleviates the problem of text label floating in contrastive learning. The method proposed in the present invention trains a text similarity discrimination model. When performing contrastive learning, the similarity between two texts obtained by data augmentation can be used as a weight to weight the information loss function of contrastive learning, thereby guiding the contrastive learning process.

[0087] (2) The contrastive learning guidance method proposed in the present invention is highly versatile. The contrastive learning guidance module proposed in the present invention is independent of the data type and is applicable to both tabular data and image data. For different data types, only the data encoding model structure needs to be modified. After the model is fitted, the contrastive learning process for that data type can be guided.

[0088] (3) Compared with the unlabeled data difficult sample filtering module proposed by the existing semi-supervised text classification algorithm, the present invention has better difficult data filtering capabilities and can help the model to be trained better in the early stage of training.

[0089] (4) The text classification model based on weighted contrastive learning proposed in this paper effectively solves the text multi-classification problem in social media information platform scenarios and news text scenarios. Under resource-constrained conditions, it demonstrates high flexibility and efficiency in utilizing unlabeled data.

[0090] The present invention has been applied to three categories of data sets, and has shown significant improvement in effect: when implemented on the iFlytek text classification data set, compared with existing methods, there is a stable 1%-4% improvement in the Macro F1-Score indicator and a 1%-3% improvement in the Micro F1-Score indicator; when implemented on the social media consulting platform data set, compared with existing methods, there is a stable 1%-15% improvement in the Macro F1-Score indicator and a 1%-7% improvement in the Micro F1-Score indicator; when implemented on the news text data set, compared with existing methods, there is a stable 1%-7% improvement in the Macro F1-Score indicator and a 1%-4% improvement in the Micro F1-Score indicator. The calculation method of Micro F1-score is to first calculate the precision, recall and F1-score of each category, and then perform a weighted average of the precision, recall and F1-score of all categories. This evaluation indicator is suitable for when the number of samples of different categories in the data set is quite different, because it regards the performance of all categories as equally important, regardless of the number of samples in the category. Macro F1-score is calculated by calculating the precision, recall and F1-score for each category separately, and then taking the arithmetic average of these values for all categories. This evaluation index is suitable for situations where all categories are equally important, because it gives equal weight to the performance of each category and is not affected by the number of category samples. Micro F1-score pays more attention to overall performance and is suitable for situations where the categories are unbalanced, while Macro F1-score pays more attention to the average performance of each category and is suitable for situations where all categories are equally important. The present invention has achieved improvements in both indicators compared to existing methods, indicating that the present invention has better precision and recall both from the perspective of a single category and from the perspective of the overall data, and can achieve better results than existing methods whether the categories are balanced or unbalanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0092] Figure 1 Schematic diagram of the problems that arise during data augmentation for difficult samples.

[0093] Figure 2 Schematic diagram of the text similarity discrimination module architecture.

[0094] Figure 3 Schematic diagram of the difficult sample filtering module architecture.

[0095] Figure 4 Schematic diagram of the text classification network structure based on weighted contrastive learning.

[0096] Figure 5 These are the comparison results of five datasets in different fields, data types, and settings applied in a specific embodiment of the present invention.

[0097] Figure 6 Generate positive and negative sample pairs for the sqb dataset.

[0098] Figure 7 Schematic diagram of difficult samples in the sqb dataset. DETAILED DESCRIPTION

[0099] The present invention will be further described in detail with reference to the following specific examples and accompanying drawings. The processes, conditions, experimental methods, etc. for implementing the present invention, except for those specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0100] The present invention provides a text multi-classification method based on weighted contrastive learning, comprising the following steps: using a labeled dataset D l Construct a positive and negative sample pair dataset D pair ; in D pair A model Φ1 is trained on the input text to predict the similarity score of the two texts; using D l Train a text encoder Φ2 and use it to calculate D l The data representation vector is clustered in the high-dimensional hidden space to obtain the set C of cluster centers; Φ2 is used to calculate D u Characterization vector, and calculate the distance between each data characterization vector and each cluster center in C; define the distance γ as the target distance, for all D u The data whose distance to the nearest cluster center is greater than γ is filtered, and the unlabeled data whose distance is less than γ is retained to obtain the filtered unlabeled dataset D u_filtered ; in D l and D u_filteredA text classifier Φ3 is trained on the dataset to effectively classify the text; the text classifier Φ3 is used to calculate the probability of the sample belonging to each category and its category. The method of the present invention has good effectiveness, robustness and label utilization.

[0101] Specifically, see the attached Figure 1 、 2 , 3, 4, the present invention performs a text multi-classification algorithm based on weighted contrastive learning according to the following steps:

[0102] Step 1: Use a labeled dataset D l Construct a positive and negative sample pair dataset D pair ;

[0103] Step 1-1: There will be a labeled dataset D l Classify according to their respective categories and obtain the data set D of each category c ={D1, D2, ..., D N};

[0104] Step 1-2: Construction of positive sample pairs. Traverse all data and set the current traversed sample as x i , determine D c Zhong and x i Similar sample set D k , in D k Randomly sample a sample x pos , we need to ensure x i with x pos Different and combined (x i , x pos ) and (x pos , x i ) has never been selected before, the label of this sample pair is set to 1;

[0105] Step 1-2-1: Traverse the labeled data set, and set the current traversed sample to be x i ;

[0106] Step 1-2-2: Determine x i The sample set D of the category k , in D k Randomly sample a sample x pos , we need to ensure x i with x pos Different and combined (x i , x pos ) and (x pos , x i ) has never been selected before. In this process, the parameter N can be set to represent the sample x iConstruct the number of positive sample pairs. After setting this parameter, just repeat steps 1-2-2N times to obtain sample x i Construct N positive sample pairs. In addition, in this process, the sample combinations may be exhausted. To avoid being unable to perform subsequent operations after exhaustion, set the random sampling process of a sample pair to be performed at most 10 times. If no sample that meets the above conditions is found after 10 times, the sampling of the sample is terminated and return to step 1-2-1 to continue traversing the labeled data set;

[0107] Step 1-2-3: Set the label of the positive sample pair obtained by screening to 1;

[0108] Step 1-2-4: Repeat steps 1-2-1 to 1-2-3 until all samples in the labeled dataset are traversed.

[0109] Step 1-3: Negative sample pair construction. Traverse all data and set the current traversed sample as x j , determine D c Zhong and x j Similar sample set D k ,, after removing D k Data set D c / D k Randomly sample a sample x neg , we need to ensure x j with x neg Different and combined (x j , x neg ) and (x neg , x j ) has never been selected before, the label of the sample pair is set to 0;

[0110] Step 1-3-1: Traverse all data and set the current traversed sample as x j ;

[0111] Step 1-3-2: Determine x j The sample set D of the category k , after removing D k Data set D c / D k Randomly sample a sample x neg , we need to ensure x j with x neg Different and combined (x j , x neg ) and (x neg , x j ) has never been selected before. Similar to the way of constructing positive sample pairs, in this process, the parameter M can be set to represent the sample x jThe number of negative sample pairs constructed. After setting this parameter, just repeat steps 1-3-2M times to obtain sample x j Construct M negative sample pairs. Similarly, to avoid exhausting the sample combinations, set the random sampling process for a sample pair to be performed at most 10 times. If no sample that meets the above conditions is found after 10 times, the sampling of the sample is terminated and return to step 1-3-1 to continue traversing the labeled dataset;

[0112] Step 1-3-3: Set the labels of the negative sample pairs obtained by screening to 0;

[0113] Step 1-3-4: Repeat steps 1-3-1 to 1-3-3 until all labeled datasets are traversed.

[0114] Step 1-4: Combine the positive sample pairs obtained in step 1-2 and the negative sample pairs obtained in step 1-3 to obtain the positive and negative sample pair dataset D pair .

[0115] Step 2: Refer to the attached Figure 2 , in the positive and negative sample dataset D pair A text similarity discrimination model Φ1 based on a twin network is trained. The model is used to predict the similarity scores of the two input texts.

[0116] Step 2-1: The positive and negative samples generated in step 1 are combined into the dataset D pair Input text similarity discrimination model Φ1;

[0117] Step 2-2: Obtain a high-dimensional representation vector for the sample pair. The input sample pair consists of two texts, text1 and text2. The text encoder E1 generates two text representation vectors v1 and v2 of the same dimension.

[0118] Step 2-3: Vector concatenation: Concatenate the two text representation vectors v1 and v2 obtained in step 2-2 to obtain the concatenated vector v3.

[0119] Step 2-4: Similarity score prediction. The concatenated vector v3 obtained in Steps 2-3 is input into a multi-layer neural network F, which is used to predict similarity scores. This network maps the high-dimensional vector v3 into a scalar value s, which serves as the prediction of the similarity score between the two input texts.

[0120] Step 2-5: Use the cross entropy loss function to make the predicted similarity score s close to the label of the input sample pair.

[0121] Step 2-6: Repeat steps 2-1 to 2-5 until the cross entropy loss function of the text similarity discrimination model Φ1 no longer decreases and the model converges.

[0122] Step 3: Refer to the attached Figure 3 , for the unlabeled dataset D u The samples that are difficult to be distinguished by the model are filtered to obtain the filtered unlabeled dataset D u_filtered ;

[0123] Step 3-1: Use the multi-class cross-entropy loss function to ensure that the predicted category of the input text matches its label. The text encoder Φ2, based on the BERT pre-trained model, maps the input raw text into a high-dimensional vector. This high-dimensional vector is then fed into a multi-layer neural network, which outputs a one-dimensional vector. Each value in the vector represents the predicted probability that the text belongs to that category. The multi-class cross-entropy loss function is then used to ensure that the predicted category of the input text matches its label. Optimization is continued until the multi-class cross-entropy loss function no longer decreases, indicating model convergence.

[0124] Step 3-1-1: Calculate text representation. The text encoder Φ2 will map the input text sample text into a high-dimensional text representation vector v;

[0125] Step 3-1-2: Label prediction. Input the text representation vector v obtained in step 3-1-1 into the multi-layer neural network G. This network maps the high-dimensional vector v into a vector logits. The dimension of the vector logits is 1×R, where R is the number of label categories. Each value in the logits represents the probability that the sample belongs to the corresponding category.

[0126] Step 3-1-3: Use the multi-classification cross entropy loss function to make the predicted logits close to the label of the input sample. The formula of the loss function is as follows:

[0127]

[0128]

[0129] Where B is the batch size. is the probability that the model predicts that sample i belongs to the kth class, is the label of sample i in the kth category. If sample i belongs to the kth category, then is 1 if the value is set, otherwise it is 0. is the probability predicted by the softmax function of logits. The goal of the text encoder model is to minimize the multi-classification cross entropy loss function.

[0130] Step 3-1-4: Repeat steps 3-1-1 to 3-1-3 until the model's multi-classification cross entropy loss function no longer decreases and the model converges. The text encoder Φ2 will be used to extract features from unlabeled data later.

[0131] Step 3-2: Find the representation of all labeled sample data. l All sample data in the text encoder Φ2 in step 3-1 are input to obtain the text representation vector V of the labeled sample data l ;

[0132] Step 3-3: The text representation vector V obtained in step 3-2 l Use the k-means algorithm to perform clustering operations and obtain the cluster centers {C1, C2, ..., C class_cnt}, where C class_cnt is the number of categories;

[0133] Step 3-4: Filter the unlabeled data. Use the text encoder Φ2 obtained in step 3-2 to filter the unlabeled dataset D u Encode each sample in to obtain the representation vector of all unlabeled data, and calculate the distance from each representation vector to the nearest cluster center, recorded as d. Set a target distance γ, retain all unlabeled data with the condition d ≤ γ, and filter out other unlabeled data;

[0134] Step 3-4-1: Traverse the unlabeled dataset D u , let the current unlabeled data be x;

[0135] Step 3-4-2: Use the text encoder Φ2 obtained in step 3-2 to encode x to obtain its representation vector v;

[0136] Step 3-4-3: Calculate v to {C1, C2, ..., C class_cnt The distance from the cluster center is calculated, and the distance to the nearest cluster center is calculated, which is recorded as d;

[0137] Step 3-4-4: Set a target distance γ. If d≤γ, retain the unlabeled data x, otherwise filter out x.

[0138] Step 3-5: Repeat steps 3-4 until all unlabeled data have been filtered, and the filtered unlabeled dataset D is obtained. u_filtered .

[0139] Step 4: Refer to the attached Figure 4 , in the labeled dataset D l And the filtered unlabeled dataset D u_filtered Train a text classifier Φ3 to effectively classify text;

[0140] Step 4-1: Initialize a text encoder E2 using the BERT pre-training model and randomly initialize a multi-layer neural network H;

[0141] Step 4-2: There will be a labeled dataset D l Input the text encoder E2 to obtain a high-dimensional representation vector of the text, then input the high-dimensional representation vector into the multi-layer neural network H, and output a one-dimensional vector. Each value in the vector represents the predicted probability that the text belongs to that category;

[0142] Step 4-3: Use the cross entropy loss function to make the predicted probability obtained in step 4-2 close to the text label;

[0143] Step 4-4: Use the filtered unlabeled dataset D u_filtered Perform contrastive learning. The contrastive learning first converts the unlabeled data sample x u Perform two data enhancements to obtain two enhanced samples a i and a j ; Then we will enhance sample a i and a j Input the text encoder E2 respectively to get the enhanced sample a i and a j Characterization vector z i and z j ;

[0144] Step 4-5: Use the weighted info nce loss function to make the enhanced sample representation vector z obtained in step 4-4 i and z j Proximity in representational space.

[0145] Step 4-6: Repeat steps 4-2 to 4-5 until the L of the text classifier Φ3 is contrastive The loss function no longer decreases and the model converges.

[0146] Step 5: Use the text classifier Φ3 to calculate the probability that the sample belongs to each category and determine the category to which the sample belongs. Use the text classifier Φ3 to calculate the probability that the sample belongs to each category and determine the category to which the sample belongs. After the model converges, the following calculation method can be used to obtain the category to which the model predicts the sample:

[0147] logits=Φ3(x)

[0148]

[0149]

[0150] l pred =argmax(softmax(logits))

[0151] Where Φ3 is the text classifier trained in step 4, x is the input text, logits represents the probability that the text belongs to each category after the text is input into the text classifier, softmax(·) is used to calculate the proportion of each value in the input vector, argmax(·) function is used to obtain the subscript index corresponding to the maximum value in the input vector, l pred It is the category to which the input text belongs predicted by the final text classification model Φ3.

[0152] Reference Attachment Figure 5 From the results of the text multi-classification algorithm model proposed in this invention on five data sets, except for the MacroF1-score indicator of the 20_newsgroups data set, which is 1% lower than the Bert model, the MacroF1-score and MicroF1-score indicators on other data sets are higher than other existing methods, indicating that the proposed model has the best effect, high text classification accuracy and low false alarm rate. Figure 5 The figures show the comparative results of the present invention applied to five datasets in different fields, data types, and settings.

[0153] Example 1

[0154] Step 1: Use the labeled data part D in the sqb dataset l Construct a positive and negative sample pair dataset D pair ;

[0155] Step 1-1: There will be a labeled dataset D l Classify according to their respective categories and obtain the data set D of each category c ={D1, D2, ..., D N The sqb dataset covers 39 categories, including catering, retail, beauty, leisure and entertainment;

[0156] Step 1-2: If Figure 6 The figure shows a schematic diagram of constructing positive and negative sample pairs for the sqb dataset. The upper half of the figure shows the construction of the positive sample pair. Traverse the labeled dataset, randomly sample a different sample for each sample in its category, ensure the combination is unique, and set the label to 1. For example, you can select the first data item from the catering category, "The Zuoting Youyuan at the Chengdong Wanda store is really delicious. My friend and I think the taste is very authentic." as the currently traversed sample, and then select the third data item in the catering category, "The coffee in this store is good, but the bread is a bit hard, and the overall traffic is not high, so it is suitable for chatting with friends." Together they form a positive sample pair, and the label is set to 1;

[0157] Step 1-2-1: Traverse the labeled data set, and set the current traversed sample to be x i ;

[0158] Step 1-2-2: Determine x i The sample set D of the category k , in D k Randomly sample a sample x pos , we need to ensure x i with x pos Different and combined (x i , x pos ) and (x pos , x i ) has never been selected before. In this process, the parameter N can be set to represent the sample x i Construct the number of positive sample pairs. After setting this parameter, just repeat steps 1-2-2N times to obtain sample x i Construct N positive sample pairs. In addition, in this process, the sample combinations may be exhausted. To avoid being unable to perform subsequent operations after exhaustion, set the random sampling process of a sample pair to be performed at most 10 times. If no sample that meets the above conditions is found after 10 times, the sampling of the sample is terminated and return to step 1-2-1 to continue traversing the labeled data set;

[0159] Step 1-2-3: Set the label of the positive sample pair obtained by screening to 1;

[0160] Step 1-2-4: Repeat steps 1-2-1 to 1-2-3 until all samples in the labeled dataset are traversed.

[0161] Steps 1-3: Figure 6 The lower half of the traversal is the construction of negative sample pairs. Traverse the labeled dataset, randomly sample a sample from the other categories for each sample, ensure the combination is unique, and set the label to 0. For example, you can select the second data item from the catering category, "Are waiters in restaurants so good at sales these days? Looking for comments." as the currently traversed sample, and then select the first data item from the beauty category, "This year's nail art trend is "simple and gentle." Compared with last year's trend, this year's style is more accepted by the public and also reflects a more masculine aesthetic trend." Together, form a negative sample pair and set the label to 0.

[0162] Step 1-3-1: Traverse all data and set the current traversed sample as x j ;

[0163] Step 1-3-2: Determine x j The sample set D of the category k , after removing D kData set D c / D k Randomly sample a sample x neg , we need to ensure x j with x neg Different and combined (x j , x neg ) and (x neg , x j ) has never been selected before. Similar to the way of constructing positive sample pairs, in this process, the parameter M can be set to represent the sample x j The number of negative sample pairs constructed. After setting this parameter, just repeat steps 1-3-2M times to obtain sample x j Construct M negative sample pairs. Similarly, to avoid exhausting the sample combinations, set the random sampling process for a sample pair to be performed at most 10 times. If no sample that meets the above conditions is found after 10 times, the sampling of the sample is terminated and return to step 1-3-1 to continue traversing the labeled dataset;

[0164] Step 1-3-3: Set the labels of the negative sample pairs obtained by screening to 0;

[0165] Step 1-3-4: Repeat steps 1-3-1 to 1-3-3 until all labeled datasets are traversed.

[0166] Step 1-4: Combine the positive sample pairs obtained in step 1-2 and the negative sample pairs obtained in step 1-3 to obtain the positive and negative sample pair dataset D pair .

[0167] Step 2: In the positive and negative sample dataset D pair A text similarity discrimination model Φ1 based on a twin network is trained. The model is used to predict the similarity scores of the two input texts.

[0168] Step 2-1: The positive and negative samples generated in step 1 are combined into the dataset D pair Input text similarity discrimination model Φ1;

[0169] Step 2-2: Obtain a high-dimensional representation vector for the sample pair. The input sample pair consists of two texts, text1 and text2. The text encoder E1 generates two text representation vectors, v1 and v2, of the same dimension. For example, if the positive sample pair ("The Zuoting Youyuan at the Chengdong Wanda store is really delicious. My friend and I think it tastes very authentic"), and "The coffee at this store is good, but the bread is a bit hard, and the overall flow of people is not high, so it's suitable for chatting with friends") is input into the text encoder E1, the representation vectors of these two texts will be obtained;

[0170] Step 2-3: Vector concatenation: Concatenate the two text representation vectors v1 and v2 obtained in step 2-2 to obtain the concatenated vector v3.

[0171] Step 2-4: Similarity score prediction. The concatenated vector v3 obtained in Steps 2-3 is input into a multi-layer neural network F, which is used to predict similarity scores. This network maps the high-dimensional vector v3 into a scalar value s, which serves as the prediction of the similarity score between the two input texts.

[0172] Step 2-5: Use the cross-entropy loss function to make the predicted similarity score s close to the label of the input sample pair. If the input is a positive sample pair, the corresponding label is 1. The cross-entropy loss function will obtain a smaller value when the similarity score predicted by the model is large, and a larger value when the similarity score predicted by the model is small. In this way, the model will learn the commonalities between similar texts and learn to output a higher score when two texts with high similarity are input. If the input is a negative sample pair, the corresponding label is 0. The cross-entropy loss function will obtain a smaller value when the similarity score predicted by the model is small, and a larger value when the similarity score predicted by the model is large. In this way, the model will learn the difference between dissimilar texts and learn to output a lower score when two texts with low similarity are input.

[0173] Step 2-6: Repeat steps 2-1 to 2-5 until the cross entropy loss function of the text similarity discrimination model Φ1 no longer decreases and the model converges.

[0174] Step 3: For the unlabeled dataset D u The samples that are difficult to be distinguished by the model are filtered to obtain the filtered unlabeled dataset D u_filtered ;like Figure 7 As shown, the SQB data contains a lot of unlabeled text, which can be difficult to extract during data augmentation due to semantic fluctuations. For example, if we replace the word "restaurant" with "cinema" and the word "dishes" with "popcorn," the semantics of the entire sentence will change from "catering" to "leisure and entertainment." Step 3 aims to remove these difficult examples as much as possible before conducting comparative learning.

[0175] Step 3-1: Use the cross-entropy loss function to match the predicted category of the input text with its label. The text encoder Φ2 is based on the BERT pre-trained model. It maps the input raw text into a high-dimensional vector. The resulting high-dimensional vector is then input into a multi-layer neural network, which outputs a one-dimensional vector. Each value in the vector represents the predicted probability that the text belongs to that category. The cross-entropy loss function is then used to match the predicted category of the input text with its label. Optimization is continued until the loss function no longer decreases and the model converges.

[0176] Step 3-1-1: Calculate text representation. The text encoder Φ2 will map the input text sample text into a high-dimensional text representation vector v;

[0177] Step 3-1-2: Label prediction. Input the text representation vector v obtained in step 3-1-1 into the multi-layer neural network G. This network maps the high-dimensional vector v into a vector logits. The dimension of the vector logits is 1×R, where R is the number of label categories. Each value in the logits represents the probability that the sample belongs to the corresponding category.

[0178] Step 3-1-3: Use the multi-classification cross entropy loss function to make the predicted logits close to the label of the input sample. The formula of the loss function is as follows:

[0179]

[0180]

[0181] Where B is the batch size. is the probability that the model predicts that sample i belongs to the kth class, is the label of sample i in the kth category. If sample i belongs to the kth category, then is 1 if the value is set, otherwise it is 0. is the probability predicted by the logits through the softmax function. The goal of the model is to minimize this loss function.

[0182] Step 3-1-4: Repeat steps 3-1-1 to 3-1-3 until the model's multi-classification cross entropy loss function no longer decreases and the model converges. The text encoder Φ2 will be used to extract features from unlabeled data later.

[0183] Step 3-2: Find the representation of all labeled data. l All the data in the text encoder Φ2 in step 3-1 are input to obtain the text representation vector V of the labeled data l ;

[0184] Step 3-3: The text representation vector V obtained in step 3-2 l Use the k-means algorithm to perform clustering operations and obtain the cluster centers {C1, C2, ..., C class_cnt}, where C alass_cnt is the number of categories; in the sqb dataset, C alass_cnt is 39;

[0185] Step 3-4: Filter the unlabeled data. Use the text encoder Φ2 obtained in step 3-2 to filter the unlabeled dataset D u Encode each sample in to obtain the representation vector of all unlabeled data, and calculate the distance from each representation vector to the nearest cluster center, recorded as d. Set a target distance γ, retain all unlabeled data with the condition d ≤ γ, and filter out other unlabeled data;

[0186] Step 3-4-1: Traverse the unlabeled dataset D u , let the current unlabeled data be x;

[0187] Step 3-4-2: Use the text encoder Φ2 obtained in step 3-2 to encode x to obtain its representation vector v;

[0188] Step 3-4-3: Calculate v to {C1, C2, ..., C class_cnt The distance from the cluster center is calculated, and the distance to the nearest cluster center is calculated, which is recorded as d;

[0189] Step 3-4-4: Set a target distance γ. If d≤γ, retain the unlabeled data x, otherwise filter out x.

[0190] Step 3-5: Repeat steps 3-4 until all unlabeled data have been filtered, and the filtered unlabeled dataset D is obtained. u_filtered .

[0191] Step 4: In the labeled dataset D l And the filtered unlabeled dataset D u_filtered Train a text classifier Φ3 to effectively classify text;

[0192] Step 4-1: Initialize a text encoder E2 using the BERT pre-training model and randomly initialize a multi-layer neural network H;

[0193] Step 4-2: There will be a labeled dataset D l Input the text encoder E2 to obtain a high-dimensional representation vector of the text, then input the high-dimensional representation vector into the neural network H, and output a one-dimensional vector. Each value in the vector represents the predicted probability that the text belongs to that category;

[0194] Step 4-3: Use the cross entropy loss function to make the predicted probability obtained in step 4-2 close to the text label;

[0195] Step 4-4: Use the filtered unlabeled dataset D u_filtered Perform contrastive learning. The contrastive learning first converts the unlabeled data sample x u Perform two data enhancements to obtain two enhanced samples a i and a j ; Then we will enhance sample a i and a j Input the text encoder E2 respectively to get the enhanced sample a i and a j Characterization vector z i and z j ;

[0196] Step 4-5: Use the weighted info nce loss function to make the enhanced sample representation vector z obtained in step 4-4 i and z j Proximity in representational space.

[0197] Step 4-6: Repeat steps 4-2 to 4-5 until the L of the text classifier Φ3 is contrastive The loss function no longer decreases and the model converges.

[0198] Step 5: Use the text classifier Φ3 to calculate the probability that the sample belongs to each category and determine the category to which the sample belongs. Use the text classifier Φ3 to calculate the probability that the sample belongs to each category and determine the category to which the sample belongs. After the model converges, the following calculation method can be used to obtain the category to which the model predicts the sample:

[0199] logits=Φ3(x)

[0200]

[0201]

[0202] l pred =argmax(softmax(logits))

[0203] Where Φ3 is the text classifier trained in step 4, x is the input text, logits represents the probability that the text belongs to each category after the text is input into the text classifier, softmax(·) is used to calculate the proportion of each value in the input vector, argmax(·) function is used to obtain the subscript index corresponding to the maximum value in the input vector, l pred It is the category to which the input text belongs predicted by the final text classification model Φ3.

[0204] The text multi-classification model proposed in this paper is implemented on the social media text dataset sqb, and the test MacroF1-score is: 70.22±0.02; the test Micro F1-score is: 69.51±0.07;

[0205] Example 2

[0206] The text multi-classification model proposed in this paper is implemented on the iFlytek text dataset, and the test MacroF1-score is 34.77±0.54; the test Micro F1-score is 57.78±0.52;

[0207] Example 3

[0208] The text multi-classification model proposed in this paper is implemented on the social media dataset sqb, and the test MacroF1-score is: 70.22±0.02; the test Micro F1-score is: 69.51±0.07;

[0209] Example 4

[0210] The text multi-classification model proposed in this paper is implemented on the news dataset tnews, and the test Macro F1-score is 52.55±0.21; the test Micro F1-score is 54.97±0.23;

[0211] Comparative Example 1

[0212] The best-performing existing model, CLENNM, was implemented on the iflytek text dataset, achieving a Macro F1-score of 32.71±0.05 and a Micro F1-score of 54.64±0.07.

[0213] Comparative Example 2

[0214] The best-performing existing model CLENNM was implemented on the social media dataset sqb, with a test Macro F1-score of 68.31±0.08 and a test Micro F1-score of 68.12±0.09.

[0215] Comparative Example 3

[0216] The best-performing existing model, CLENNM, was implemented on the news dataset tnews, with a Macro F1-score of 51.25±0.11 and a Micro F1-score of 52.50±0.10.

[0217] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the appended claims.

Claims

1. A text multi-classification method based on weighted contrastive learning, characterized in that: The steps include: Step 1: Use a labeled dataset D l Construct a positive and negative sample pair dataset D pair ; Step 2: In the positive and negative sample dataset D pair A text similarity discrimination model Φ1 is trained to predict the similarity scores of the two input texts; Step 3: For the unlabeled dataset D u Filter the difficult samples in the dataset to obtain the filtered unlabeled dataset D u _filtered ; Step 4: In the labeled dataset D l And the filtered unlabeled dataset D u_filtered Train a text classifier Φ3 to classify the text; Step 5: Use the text classifier Φ3 to calculate the probability that the sample belongs to each category and determine the category to which the sample belongs.

2. The text classification method according to claim 1, wherein: In step 1, the positive and negative sample data set D pair Includes positive sample pairs and negative sample pairs; each sample pair includes two text data and the label corresponding to the sample pair; The step 1 further comprises the following steps: Step 1-1: Data classification: There will be a labeled dataset D L Classify them according to their respective categories and obtain the data sets of each category. The data sets of each category are expressed as follows: D c ={D1, D2, ..., D N }; Step 1-2: Construction of positive sample pairs: traverse all the data in the labeled dataset, and set the currently traversed sample as x i , determine D c Zhong and x i Similar sample set D k , in D k Randomly sample a sample x from pos , we need to ensure x i with x pos Different and combined (x i , x pos ) and (x pos , x i ) has never been selected before, the label of this sample pair is set to 1 as a positive sample pair; Step 1-3: Negative sample pair construction: traverse all the data in the labeled dataset, and set the currently traversed sample as x j , determine D c Zhong and x j Similar sample set D k ,, after removing D k Data set D c / D k Randomly sample a sample x from neg , we need to ensure x neg with x j Different and combined (x j , x neg ) and (x neg , x j ) has never been selected before, the label of the sample pair is set to 0 as a negative sample pair; Step 1-4: Combine the positive sample pairs obtained in step 1-2 and the negative sample pairs obtained in step 1-3 to obtain the positive and negative sample pair dataset D pair .

3. The text classification method according to claim 1, wherein: In step 2, the text similarity discrimination model Φ1 is constructed based on a twin network, including a text encoder E1 based on a BERT pre-trained model and a multi-layer neural network F that outputs similarity scores; the original text data is mapped into a high-dimensional vector, and then the high-dimensional vector is mapped into a numerical scalar; the text similarity discrimination model Φ1 converges by minimizing the cross entropy loss function; The step 2 further comprises the following steps: Step 2-1: The positive and negative samples generated in step 1 are combined into the dataset D pair Input the original text similarity discrimination model Φ1; Step 2-2: The sample pair includes two texts text1 and text2. The input text encoder F1 will obtain two text representation vectors v1 and v2 of the same dimension; Step 2-3: Concatenate the two text representation vectors v1 and v2 obtained in step 2-2 to obtain the concatenated vector v3, which is the high-dimensional representation vector of the sample pair; Step 2-4: Input the concatenated vector v3 obtained in step 2-3 into the multi-layer neural network F for predicting similarity scores, and map it into a scalar value S as the prediction of the similarity score of the two input texts; Step 2-5: Minimize the cross entropy loss function so that the similarity score S predicted by the multi-layer neural network is close to the label of the input sample pair; Step 2-6: Repeat steps 2-1 to 2-5 until the cross entropy loss function of the text similarity discrimination model Φ1 no longer decreases, the text similarity discrimination model converges, and the optimized text similarity discrimination model is obtained.

4. The text classification method according to claim 1, wherein: In step 3, we use the labeled dataset D L Train a text encoder Φ2 and use Φ2 to train a labeled dataset D l Perform feature extraction to obtain the representation vectors of each data, and cluster the representation vectors in the high-dimensional hidden space to obtain the set C of cluster centers; use Φ2 to classify the unlabeled data set D u Perform representation extraction on the dataset to obtain the representation vectors of all unlabeled data, and calculate the distance between each unlabeled data representation vector and each cluster center in C; define the distance γ as the target distance, filter all unlabeled data whose distance to the nearest cluster center is greater than γ, and retain the unlabeled data whose distance is less than γ, and obtain the filtered unlabeled data set D u _filtered .

5. The text classification method according to claim 4, wherein: The step 3 further comprises the following steps: Step 3-1: Use the multi-classification cross entropy loss function to match the predicted category of the input text with its label and construct the text encoder Φ2; Step 3-2: There will be a labeled dataset D l All the data in the text encoder Φ2 in step 3-1 are input to obtain the text representation vector V of the labeled data l ; Step 3-3: The text representation vector V obtained in step 3-2 l Use the k-means algorithm to perform clustering operations and obtain the cluster centers {C1, C2, ..., C class_cnt }, where C class_cnt is the number of categories; Step 3-4: Use the text encoder Φ2 obtained in step 3-2 to train the unlabeled dataset D u Encode each sample in to obtain the representation vector of all unlabeled data, calculate the distance from each representation vector to the nearest cluster center, recorded as d; set a target distance γ, retain all unlabeled data with the condition d≤γ, and filter out other unlabeled data; Step 3-5: Repeat steps 3-4 until all unlabeled data have been filtered, and the filtered unlabeled dataset D is obtained. u_filtered .

6. The text classification method according to claim 1, wherein: In step 4, the training of the text classifier Φ3 includes two parts: using the labeled dataset D L Perform supervised training of the model and use the filtered unlabeled dataset D u _filtered Perform unsupervised training; The step 4 further comprises the following steps: Step 4-1: Initialize a text encoder E2 using the BERT pre-training model and randomly initialize a multi-layer neural network H; Step 4-2: There will be a labeled dataset D L Input the text encoder E2 to obtain a high-dimensional representation vector of the text, then input the representation vector into the multi-layer neural network H, and output a one-dimensional vector. Each value in the vector represents the predicted probability that the text belongs to that category; Step 4-3: Using L cls The loss function makes the predicted probability obtained in step 4-2 close to the label of the text; Step 4-4: Use the filtered unlabeled dataset D u_filtered Perform contrastive learning and convert the unlabeled data sample x u Perform two data enhancements to obtain two enhanced samples a i and a j ; Then we will enhance sample a i and a j Input the text encoder E2 respectively to get the enhanced sample a i and a j Characterization vector z i and z j ; Step 4-5: Using L contrastive The loss function makes the enhanced sample representation vector z obtained in step 4-4 i and z j Proximity is achieved in representational space; Step 4-6: Repeat steps 4-2 to 4-5 until the overall objective function of the text classifier Φ3 no longer decreases and the model converges.

7. The text classification method according to claim 6, wherein: In step 4-5, the L contrastive The loss function is expressed as follows: Loss Function Among them 1 [k≠i] ∈{0, 1} is the indicator function, τ is the temperature coefficient, z i is the i-th sample, z j is the sample generated by data enhancement of the i-th sample, z k It represents all samples in the same batch and their data-enhanced sample sets except z i Samples selected from outside; sg (z i , z j ) is z i and z j The similarity score output by the comparative learning guidance module.

8. The text classification method according to claim 1, wherein: In step 5, the calculation method for determining the category to which the sample belongs is as follows: logits=Φ3(x) l pred =argmax(softmax(logits)) Where Φ3 is the text classifier trained in step 4, x is the input text, logits represents the probability that the text belongs to each category after the text is input into the text classifier, softmax(·) is used to calculate the proportion of each value in the input vector, argmax(·) function is used to obtain the subscript index corresponding to the maximum value in the input vector, l pred It is the category to which the input text belongs predicted by the final text classification model Φ3.

9. A multi-classification system for implementing the text multi-classification method according to any one of claims 1 to 8, characterized in that: The system includes a positive and negative sample pair construction module, a text similarity discrimination module, an unlabeled difficult sample filtering module, and a semi-supervised text multi-classification module; wherein, The positive and negative sample pair construction module is used to generate a positive and negative sample pair dataset using labeled data, and use the positive and negative sample pair dataset to train a text similarity discrimination model; The text similarity discrimination module is used to guide the subsequent comparative learning of unlabeled data; The unlabeled difficult sample filtering module is used to filter difficult samples to reduce the interference of difficult samples on the model in the early stage of training; The semi-supervised text multi-classification module is used to integrate the text similarity judgment module and the unlabeled difficult sample filtering module to achieve multi-classification of text.

10. Application of the text classification method according to any one of claims 1 to 8, or the multi-classification system according to claim 9, in news text classification and social media dynamic text classification.

Images