Highly unbalanced text classification-oriented enhanced contrast learning method and device

Abstract

The invention discloses an enhanced contrast learning method and device for highly unbalanced text classification, and the method comprises the steps: carrying out the preprocessing of text data in a training data set and a test data set, and obtaining corresponding original training texts and test texts; respectively inputting the original training text and the test text into the RoBERTa model to obtain corresponding text embedding representations; constructing a text classification model based on data enhancement and comparative learning, wherein the text classification model comprises a text enhancement module, a text representation module, a comparative learning module and a text classification module; and applying the trained and tested text classification model to classification processing of to-be-processed text data, and outputting a classification result. The invention provides a new self-constrained stacked data enhancement method and a new double-weighted contrast loss function, the characterization problem caused by data imbalance and data lack can be solved, and a better text classification effect is achieved.

Description

Technical Field

[0001] The present invention relates to the fields of text mining, natural language processing and deep learning technology, and in particular to an enhanced contrastive learning method and device for highly imbalanced text classification. Background Art

[0002] With the development of neural networks and deep learning technology, the distribution representation of words has been widely used in many natural language processing tasks. Through the representation learning of word distribution, the smallest semantic unit in the text - words (or characters) are converted into dense vectors that can be recognized and calculated by computers through a series of transformations. Therefore, the representation learning model is also called the word vector model. A series of word vector models such as Word2vec, GloVe, ELMo, and BERT have emerged. Among them, the BERT model, which obtains context information through the attention mechanism and multi-head mechanism, has significantly improved the performance of multiple natural language processing tasks and is a milestone in this field.

[0003] However, due to the complexity of text data, there are still many challenges in actual text classification applications. Even if text representation based on models such as BERT is used, ideal text classification performance cannot be obtained. For example, one of the most common problems is the imbalanced distribution of text data among categories. That is, some categories become majority categories because there are more labeled data; while other categories become minority categories because the data is difficult to obtain or difficult to manually label, there are only less labeled data. When there are very few labeled data in the minority category, it is highly unbalanced data, which makes the data have both imbalance and data shortage problems. Learning classification from such data is a challenging task. Due to the imbalanced number of data for each class, it will have a serious adverse effect on the performance of the classifier. Classifiers trained with unbalanced data tend to predict the majority class. Because it ignores the influence of the minority class and confuses it with the majority class, it is difficult to learn a robust semantic representation of the text.

[0004] In the field of deep learning, many approaches have emerged to address this problem, such as incorporating label interactions into model construction and designing customized loss functions. Some studies advocate expanding the representation space by introducing statistical correlations in graph neural networks in the projection head. Meanwhile, other approaches modify the traditional binary cross entropy (BCE) by assigning higher weights to certain samples and labels, or introduce auxiliary loss functions for regularization. In particular, recent methods based on supervised contrastive learning as auxiliary losses have successfully enhanced semantic representations in text classification.

[0005] Although contrastive learning is an effective method, its application in imbalanced text classification remains challenging. First, selecting effective data augmentation techniques in contrastive learning of text data has proven to be a challenging task. For image data, data augmentation can be easily achieved using various geometric transformations; unlike images, the application of general data augmentation techniques in natural language processing is more difficult due to the discreteness, complexity, and poor anti-interference ability of natural language. Second, when the distribution of text data shows an imbalanced or long-tail pattern, some labels are significantly more common than others, which will greatly reduce the quality of the representation. Since there are only a few or very few labeled data for instances of minority categories, the "lack of positive pairs" problem will arise when constructing positive and negative pairs in contrastive learning, which directly affects the effect of contrastive learning. Summary of the invention

[0006] Technical purpose: In response to the above problems, the present invention proposes a new self-constrained stacked data enhancement method and a new dual weighted contrast loss function, which can solve the representation problems caused by data imbalance and data lack, and achieve better text classification effects.

[0007] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution:

[0008] An enhanced contrastive learning method for highly imbalanced text classification includes the following steps:

[0009] S1, dividing the collected text data set into a training data set and a test data set, preprocessing the text data in the training data set and the test data set respectively to obtain the corresponding original training text and test text;

[0010] S2, input the original training text and test text into the RoBERTa model respectively to obtain the corresponding text embedding representation;

[0011] S3, constructing a text classification model based on data enhancement and contrastive learning, wherein the text classification model includes a text enhancement module, a text representation module, a contrastive learning module and a text classification module, using the original training text and its embedded representation to train the text classification model, and using the embedded representation of the test text to test the trained text classification model;

[0012] S4. Apply the trained and tested text classification model to the classification of the text data to be processed, and output the classification result of the text data to be processed.

[0013] Preferably, in step S1, the preprocessing operations include sentence segmentation, word segmentation, stop word removal and punctuation removal;

[0014] The original training dataset is denoted as D orig ,

[0015] Among them, x i is the i-th text in the training dataset, y i is the type label corresponding to the i-th text, and N1 is the number of texts in the training data set;

[0016] In step S2, RoBERTa obtains the input text x i The word vector embedding representation E(x i ).

[0017] Preferably, in step S3, the text enhancement module includes a text enhancer based on a masked language model and a text enhancer based on perturbations, and the text enhancer based on the masked language model proposes a data enhancement algorithm for dynamically randomly selecting a mask. The text in is masked, and then the text is reconstructed by fine-tuning the RoBERTa model, outputting a set of text embedding representations of N2 is x i Corresponding enhanced text;

[0018] Perturbation-based text augmentation uses adversarial training to augment text collections. Add adversarial perturbations and output a set of N2 enhanced data embedding representations yes Corresponding enhanced text;

[0019] The text enhancement module designs a consistent self-constrained loss function L consis :

[0020]

[0021] Among them, L BCE is the loss function of the text classification module, R con To introduce the label consistency loss described by KL divergence, δ and ξ are the corresponding loss weight coefficients

[0022] Preferably, the data enhancement algorithm processing flow of the dynamic random selection mask is as follows:

[0023] Obtain a sentence obtained through the word segmentation operation in preprocessing, where the sentence is represented as a set of multiple words;

[0024] Randomly select words in the masked sentence at a preset ratio, and replace the masked words with randomly generated masks to obtain the masked sentence;

[0025] Merge consecutive mask tokens to get the merged sentence.

[0026] Preferably, in step S3, the text x i The word vector embedding representation E(x i ) and word embedding representation of enhanced text Constitute the total text representation set Input text representation module;

[0027] The text representation module obtains long-distance dependency information through a multi-head attention mechanism and outputs the feature vector representation H(x) of the text, where H(x) = [h1,h2,h3,…,h j ,…,h n ].

[0028] Preferably, in step S3, the comparative learning module performs the following process:

[0029] Construct positive and negative sample pairs for contrastive learning: Given a training batch with M samples, for each text d in the training batch i , construct positive sample pairs p for contrastive learning pos =(d i ,d j(i) ), that is, with the same label y i The sample pairs can be obtained by M1 is the number of positive sample pairs; for each text d in the training batch i , construct negative sample pairs p for contrastive learning neg =(d i ,d k ), that is, sample pairs with different labels, we can get a set of negative sample pairs M2 is the number of negative sample pairs; then the loss function of classic contrastive learning can be obtained:

[0030]

[0031] Among them, the temperature coefficient τ is a hyperparameter used to adjust the degree of penalty for the model to focus on difficult negative samples;

[0032] Design a doubly weighted contrastive learning loss function L DWCL :

[0033]

[0034] W1 and W2 are weight functions;

[0035]

[0036] 0<μ<1, l(d i ) is the loss of each sample in the training batch;

[0037] The text representation module improves the feature space distribution by constructing positive and negative sample pairs and a double weighted contrast learning loss function training strategy, and outputs the optimized text vector representation H(x i ).

[0038] Preferably, in step S3, the text classification module inputs the feature vector H(x) output by the text representation module, and designs a cross entropy loss function L BCE , use the SoftMax function to classify the text and output the probability of text classification;

[0039] The consistency self-constraint loss function L consis , double weighted contrastive learning loss function L DWCL And the cross entropy loss function L BCE Together they constitute the total loss function L of the text classification model based on data enhancement and contrastive learning:

[0040] L=L BCE +αL consis +βL DWCL

[0041] Among them, α and β are the corresponding loss weight coefficients;

[0042] According to the total loss function L, the total loss of the model is calculated and derived, the error is back-propagated, the model parameters are optimized, and the model training is completed.

[0043] An enhanced contrastive learning device for highly imbalanced text classification, comprising:

[0044] The data preprocessing module is used to divide the collected text data set into a training data set and a test data set, and preprocess the text data in the training data set and the test data set respectively to obtain the corresponding original training text and test text;

[0045] The RoBERTa module is used to input the original training text and test text into the RoBERTa model to obtain the corresponding text embedding representation;

[0046] A model construction module is used to construct a text classification model based on data enhancement and contrastive learning, wherein the text classification model includes a text enhancement module, a text representation module, a contrastive learning module and a text classification module. The text classification model is trained using the original training text and its embedded representation, and the trained text classification model is tested using the embedded representation of the test text.

[0047] The classification processing module is used to apply the text classification model that has completed training and testing to the classification processing of the text data to be processed, and output the classification results of the text data to be processed.

[0048] A computer-readable storage medium stores a computer program or instruction, which implements the steps of the method when executed by a processor.

[0049] A computer program product comprises a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the steps of the method are implemented.

[0050] Beneficial effects: The present invention proposes a new text classification method based on data enhancement and contrastive learning. Its innovations compared with the prior art are mainly reflected in the following three aspects:

[0051] (1) The present invention uses data enhancement and proposes a new self-constrained sequential stacking data enhancement method to generate minority category data and solve the problems of imbalance and data shortage. In this data enhancement method, the masked language model-based method and the adversarial perturbation-based method are combined through the sequential stacking method to enhance the embedding representation of the text; and by constructing a consistency loss function and setting a reasonable range of noise, it is ensured that the enhanced features not only retain most of the distribution of the original data, but also approach the real distribution of the feature space, thereby effectively improving the performance of text classification.

[0052] (2) The present invention utilizes contrastive learning to design a new dual-weighted contrast loss function. By constructing two weight functions, the weights of positive samples are improved and the effect of positive samples being pulled closer and negative samples being pushed away is strengthened, thereby improving the feature space distribution and optimizing the data representation.

[0053] (3) The present invention constructs a universal and high-performance Chinese text classification model, which has high generalization ability and can be applied to different multi-class Chinese text classification scenarios, especially when the data is highly unbalanced or lacking, and has good robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flowchart of an enhanced contrastive learning method for highly imbalanced text classification proposed by the present invention;

[0055] Figure 2 for Figure 1 Schematic diagram of the structure of the text classification model based on data enhancement and contrastive learning;

[0056] Figure 3 This is an example diagram of an MLM-based text enhancer;

[0057] Figure 4 Schematic diagram of the structure of the multi-head attention representation module. DETAILED DESCRIPTION

[0058] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0059] In view of the problems of data imbalance and data shortage in highly unbalanced text multi-class classification: that is, the sample data is unevenly distributed among categories, and the number of samples in minority categories is extremely small, which makes it very difficult to correctly classify. The present invention proposes a text classification model based on data enhancement and contrastive learning. This method first generates minority category data through a self-constrained stacking data enhancement method to alleviate the problem of data shortage; and then further optimizes data representation on the basis of data enhancement through the design of a double weighted contrast loss function.

[0060] The present invention enhances the data representation of text, not only retains the distribution of most of the original data and approaches the real distribution of the feature space, but also improves the distribution of the feature space through contrastive learning, thereby improving the performance of text classification.

[0061] First, the present invention proposes a new self-constrained stacking data enhancement method to generate minority category data to solve the problem of "lack of positive pairs". In this data enhancement method, the data enhancements generated by the Masked Language Model (MLM) and the perturbation method based on are combined through a sequential stacking method to increase the diversity of the enhanced text; and by constructing a consistency loss function, a reasonable range of noise is set for the generated text data to ensure that important information is not lost and thus maintain label invariance.

[0062] Secondly, the present invention designs a new dual-weighted contrast loss function to effectively solve two important difficulties in contrastive learning. By setting weights for positive samples, the ratio between labels in the positive and negative pairs is improved, and the lack of positive samples is further corrected; at the same time, by adding a learning coefficient to the contrastive loss, the effect of pulling positive sample pairs (texts belonging to the same category) closer and negative sample pairs (texts belonging to different categories) farther apart is strengthened, reducing the intra-class distance while increasing the inter-class distance, improving the feature space distribution. By using dual weight adjustment, on the basis of data enhancement, contrastive learning can achieve the purpose of optimizing data representation.

[0063] like Figure 1 and Figure 2 As shown, the model proposed in the present invention is divided into four main modules: (1) text enhancement module; (2) contrastive learning module; (3) text classification module; (4) text representation module, wherein the contrastive learning and text classification models share the representation module.

[0064] For a given input original data set D orig A text x i =(w1, w2, ..., w n), the output of text classification is to obtain x i A class label y i , where w i Is the text x i A Chinese character (word or token) in i Indicates the type label corresponding to the text. The length of the text is n, y i ∈{1,2,…L} has L types of multi-class text labels. The original data set contains N1 texts, namely The present invention uses RoBERTa to obtain the input text x i The word vector embedding representation E(x i ), the RoBERTa pre-training model is based on BERT and has been optimized and improved. It adopts a dynamic masking strategy, which makes the model rely more on context to predict masked words and can obtain semantic information more fully. orig For each text, we can get the embedding representation set

[0065] 1. Text enhancement module:

[0066] This module consists of a masked language model (MLM)-based text enhancer and a perturbation-based text enhancer, and generates an embedded representation of new text through sequential stacking. By constructing a consistency loss function and setting a reasonable range of noise for the newly generated text, it ensures that important information is not lost or changed, thereby maintaining the invariance of the generated text label.

[0067] (1) Sequentially stacked data enhancement: Since the masked language model-based and adversarial training-based data enhancements have the same goal of generating neighbors around the original training samples, the only difference is the way of generating adjacent data. Therefore, the present invention combines two different data enhancements in a sequential stacking manner so that they can complement each other and further increase the diversity of the data. It is worth noting that due to the discreteness of text data, some stacking orders are not feasible. For example, it is unreasonable to exchange the order of combining the two methods and provide an adversarially perturbed embedding sequence to the MLM-based text enhancer. Without loss of generality, the present invention chooses to superimpose adversarial training on the MLM-based text enhancer to achieve a sequential stacking operation.

[0068] (2) MLM-based text enhancer: In the MLM data enhancement module, the present invention proposes a data enhancement algorithm DSMLM-DA based on dynamic random selection (Dynamic & Selective) mask, and obtains text embedding enhancement by fine-tuning the RoBERTa model. Unlike other MLM-based text enhancement methods, the present invention does not actually generate enhanced text data, but directly obtains its RoBERTa-based data embedding representation as the input of subsequent modules.

[0069] The specific steps of DSMLM-DA are shown in Algorithm 1. For the input text, a word segmentation tool (such as Jieba) is used to segment all sentences to obtain x i =(p1, p2, ... p m ), m is the number of participles in the sentence. Through participles, D orig All sentences in get k different word segmentations.

[0070]

[0071]

[0072] At the beginning of fine-tuning, we randomly mask words in the sentence at a ratio of e% (assuming e% = g). e is a random number from a Gaussian distribution N(μ, σ 2 ), where μ is D orig The mean of all sentences in contains different number of word segments, and the variance σ is set to 1 / k. That is, in the generation stage, a dynamic mask rate is randomly selected in different training rounds using Gaussian distribution to increase the diversity of generated data. Through this step, the masked sentence is obtained:

[0073] x=(p1,Mask1,p3,p4,Mask2,Mask3,p7,…,Mask g ,p m ),

[0074] Mask j (j=1,…,g) is a randomly generated mask, where j is a random number in [1,m].

[0075] Then, for the masked sentence, merge the consecutive mask tokens to get the merged sentence:

[0076] x=(p1,Mask1,p3,p4,Mask 2-3 ,p7,…,Mask g ,p m ).

[0077] Finally, the text reconstruction task is solved by fine-tuning the RoBERTa model. Given a damaged (masked) sentence, RoBERTa learns to restore the entire linearized new sentence and achieves data enhancement of the text by replacing the masked words in the original sentence. Here, in order to ensure the diversity of the generated text, the tth most likely component of the probability distribution is selected as the embedding representation of the newly generated text. Instead of selecting the most likely one as the output. After repeating N2 times, we get a set of N2 enhanced texts. Figure 3 An example for t=3 is shown.

[0078] (3) Perturbation-based text enhancer: This paper simultaneously uses adversarial training to generate perturbations and enhance text data embedding to improve the robustness of the model. Compared with MLM-based data enhancement techniques, adversarial training does not require domain knowledge to generate additional training examples.

[0079] In this step, the present invention uses the classic adversarial training method to add adversarial perturbations, that is, adding perturbations to the representation after data enhancement. In the word embedding representation vector, a new embedding representation is generated The following perturbation is generated for the word embedding representation vector:

[0080]

[0081] Where z(·) represents the predicted output of the text classification module (Formula 17), L BCE is the loss function for training the neural network of this submodule. R AT When the formula is used for network training, the loss function L BCE Back propagation, calculate the required gradient, and generate small perturbations. This method is called the fast gradient method for generating adversarial representations. Here, ∈ is a model hyperparameter that adjusts the magnitude of the perturbation.

[0082] Word Embedding Representation Vector After mixing the perturbations, an enhanced representation vector is generated

[0083]

[0084] At this point, the enhanced text can be obtained The embedding representation set

[0085] (4) Consistency self-constraint loss function:

[0086] After the data enhancement is stacked in the above order, for an original training data (E(x i ),y i ), after MLM-based text enhancement, we get the text embedding representation After the adversarial disturbance processing, the The enhanced data embedding representation of is through E(x i ) Implement the following two different data enhancements to obtain:

[0087]

[0088] In order to constrain the new data noise generated during the two transformations within a reasonable range and ensure label invariance, the present invention introduces the label consistency loss R described by KL divergence. con :

[0089]

[0090] in, p θ (E(x i )) represents text classification, that is, text embedding E(x i ) is the input to get the text classification label (so that H(x i )=E(x i )), θ represents the parameters in the classification module (i.e., the parameters in Formula 17).

[0091] Then the self-constrained consistency training loss function of the sequential stacking operation can be expressed as follows:

[0092]

[0093] Among them, the first term corresponds to the cross entropy loss term of the classification module, and the second term is the adversarial loss term, R con express The consistency loss term between them, δ and ξ are its loss weight coefficients.

[0094] 2. Multi-Head Attention Representation Module

[0095] In the text classification model of deep learning, the attention mechanism is an important feature representation method to achieve different levels of attention to different elements. The multi-head attention mechanism is very good at learning long-distance dependencies and expands the model's ability to pay attention to different positions. This paper uses multi-head attention to enhance text representation. Through the original text and its enhancement, their embedded representations E(x i )and These are the inputs for subsequent contrastive learning and text classification. Without loss of generality, they are uniformly represented as text embedding vectors E(x i )=[e1,e2,e3,…,e n ], that is, the total text representation set is obtained

[0096] like Figure 4 In the MHA module shown in the figure, the embedding vector E(x i ) as input, and obtain long-distance dependency information through the multi-head attention mechanism. Each pair of positions in the multi-head attention mechanism is directly connected, and each position j will pay attention to all the positions before it and obtain a set of attention scores. The obtained MHA represents h j The representation is as follows:

[0097] h j =MHA(e1,e2,…,e j )=Concat(head1(e j ),head2(e j ),…,head c (e j ))W (Formula 7)

[0098] Among them head c (e j ) is the result of the cth attention head, W is the linear mapping matrix, and each attention head is a different weight matrix operation:

[0099]

[0100] The attention head weights are normalized using the SoftMax function:

[0101]

[0102] Among them, query, key i and value i are the corresponding vectors obtained by linearly projecting the input vector into the query, key, and value spaces, respectively. k is the dimension of the key vector. The specific calculation formula is as follows:

[0103] query,key i ,value i =W q e j ,W k e i ,W v e i (Formula 10)

[0104] Where W q , W k and W v is the mapping matrix, and different attention heads correspond to different mapping matrices.

[0105] Through the multi-head attention representation module, the text is obtained in the form of H(x) = [h1,h2,h3,…,h n ] is represented by the feature vector.

[0106] 3. Comparative Learning Module

[0107] The present invention is aimed at supervised multi-classification tasks, and the features H(x) extracted by the above text representation module are as follows: n ], by using contrastive learning, we construct positive and negative sample pairs and a double weighted contrastive learning loss function training strategy to improve the feature space distribution and further optimize the data representation.

[0108] (1) Construct positive and negative sample pairs for contrastive learning: A training batch with a sample size of M is extracted from the training batch, and for each text d in the training batch i , construct positive sample pairs p for contrastive learning pos =(d i ,d j(i) ), that is, with the same label y i The sample pairs can be obtained by M1 is the number of positive sample pairs; for each text d i , construct negative sample pairs p for contrastive learning neg =(d i ,d k ), that is, sample pairs with different labels, we can get a set of negative sample pairs M2 is the number of negative sample pairs; usually M2>M1. Then we can get the loss function of classic contrastive learning:

[0109]

[0110] Among them, the temperature coefficient τ is a hyperparameter used to adjust the degree of penalty for the model to focus on difficult negative samples, and sim() is the cosine similarity function.

[0111] (2) Double Weighted Contrastive Learning Loss Function

[0112] The present invention designs a new dual-weighted contrast loss function for contrastive learning, which uses the label-based mutual relationship to weight the positive sample pairs, and considers the imbalance of labels in the positive and negative sample pairs (M2>M1) for weighting. On the basis of data enhancement, the sample imbalance is further corrected. The dual-weighted contrast loss in the present invention can be defined as follows:

[0113]

[0114] l(d i ) is the loss of each sample in the training batch, then L DWCL Average loss. The weight function W1 assigns equal weights to all sufficiently similar positive sample pairs and smaller weights μ(0<μ<1) to all other sample pairs. W1 strengthens the effect of pulling positive sample pairs closer and pushing negative sample pairs farther away by adding coefficients to the contrast loss, thereby improving the distribution of feature space. It is defined as follows:

[0115]

[0116] The weight function W2 represents the weight between samples, improves the lack of positive samples, and makes the equation consistent with Jaccard similarity in the case of label balance. N(i,j) is its normalization term, which is defined as follows:

[0117]

[0118] 4. Text Classification Module

[0119] Finally, by contrastive learning and improving the feature space distribution, the text representation module can obtain a better text vector representation to obtain an optimized text vector representation H(x i ): The smaller the text vector distance, the closer the semantic information is; and the larger the text vector distance, the farther the semantic information is, which can be better applied to text classification tasks.

[0120] The output matrix H of the multi-head attention representation module is obtained through the fully connected layer to obtain the matrix z = u × H + b, where u and b represent the weight matrix and bias in the fully connected layer respectively, and the output of the i-th node is z i , there are L output nodes in total.

[0121] z(H(x i ))=u×H(x i )+b(Formula 17)

[0122] The SoftMax function (normalized exponential function) is used to classify text. The SoftMax expression is shown as follows:

[0123]

[0124] Among them, p i It represents the probability output of the text being classified into category i, and L is the total number of categories for text classification.

[0125] The target loss function used is the classic cross entropy loss (BCE) loss function, and the formula is as follows:

[0126]

[0127] Among them, p is the estimated distribution value of the sample, q is the true distribution value, and H(p,q) is the cross entropy between p and q.

[0128] 5. The total loss function of the model

[0129] The model adopts a multi-task training mode, which consists of three learning tasks: self-constrained consistency loss L consis , double weighted contrast loss L DWCL and the cross entropy loss L for text classification BCE The total loss function formula is as follows:

[0130] L=L BCE +αL consis +βL DWCL (Formula 20)

[0131] Among them, α and β are the loss weight coefficients of the learning task.

[0132] The specific training process of the Chinese text classification model in the present invention ( Figure 2 The black arrow in the figure is as follows:

[0133] 1. For the training set Perform text preprocessing (including sentence segmentation, stop word removal, and punctuation removal);

[0134] 2. Construct MLM text enhancement: Use the enhancement algorithm DSMLM-DA proposed in this paper to obtain text data enhancement by fine-tuning the RoBERTa model;

[0135] (1) Through the word segmentation operation in preprocessing, we get x i =(p1, p2, ... p m ), m is the number of words in the sentence. Through word segmentation, D orig All sentences in get k different word segmentations.

[0136] (2) Randomly mask the words in the sentence at a ratio of e% (assuming e% = g). e is a word from a Gaussian distribution N(μ, σ 2 ), where μ is the mean of the number of word segments in all sentences and the variance σ is set to 1 / k.

[0137] (3) Obtain the masked sentence x = (p1, Mask1, p3, p4, Mask2, Mask3, p7, ..., Mask g ,p m ), where Mask j (j=1,…,g) is a randomly generated mask.

[0138] (4) Merge consecutive mask tokens to obtain the merged sentence

[0139] x=(p1,Mask1,p3,p4,Mask 2-3 ,p7,…,Mask g ,p m ).

[0140] (5) Fine-tuning RoBERTa: Select the tth most likely component of the probability distribution as the newly generated text embedding representation By replacing words in the original sentence, text data enhancement is achieved.

[0141] (6) After repeating (1)-(5) N2 times, we get a set of N2 enhanced text embedding representations.

[0142] 3. Construct text enhancement based on adversarial perturbations:

[0143] (1) Using the adversarial loss function L BCE Back propagation, calculating the required gradients, generating perturbations

[0144]

[0145] (2) Word Embedding Representation Vector After mixing the perturbations, the enhanced representation vector is generated using Formula 2

[0146] (3) Enhanced text The embedding representation set

[0147] 4. Calculate the consistency self-constraint loss: After the data enhancement is stacked in the above order, the noise generated during the two transformations is constrained to be within a reasonable range to ensure label invariance. Use Formula 6 to calculate the consistency self-constraint loss L consis .

[0148] 5. Use RoBERTa to get the input text x i The word vector representation E(x i ), and get the embedding representation set of each text

[0149] 6. Merge the original text and the enhanced text: Through the original text and its enhancement, their embedding representations E(x i )and These are the inputs for subsequent contrastive learning and text classification. Without loss of generality, they are uniformly represented as an embedded vector form E(x i )=[e1,e2,e3,…,e n ] to get the total text representation set

[0150] 7. Multi-head attention representation: The embedding vector E(x i ) as input, and obtain long-distance dependency information through the multi-head attention mechanism. Using Formula 7, we get the form H(x i )=[h1,h2,h3,…,h n ] is the feature vector of ].

[0151] 8. Constructing contrastive learning

[0152] (1) Constructing positive and negative sample pairs for contrastive learning: In a given training batch of sample size M, for each text d i , construct positive sample pairs p for contrastive learning pos =(d i ,d j(i) ), that is, with the same label y i The sample pairs can be obtained by M1 is the number of positive sample pairs; for each text d i , construct negative sample pairs p for contrastive learning neg =(d i ,d k ), that is, sample pairs with different labels, we can get a set of negative sample pairs M2 is the number of negative sample pairs; usually M2>M1.

[0153] (2) Calculate the double weighted contrastive learning loss: Use formula 13 to calculate the loss L DWCL .

[0154] 9. Build a text classification module:

[0155] (1) Use the SoftMax function (normalized exponential function) to classify text and use Formula 18 to obtain the sample estimated distribution value.

[0156] (2) Using formula 19 and the true distribution value of the sample, calculate the cross entropy loss function L BCE .

[0157] 10. Calculate the total loss of the model

[0158] (1) The model adopts a multi-task training mode. The training process consists of three learning tasks. Formula 20 is used to calculate the total model loss.

[0159] (2) Based on the derivation of the total loss of the model, the error is back-propagated to optimize the model parameters.

[0160] The specific test (category labeling) process of the Chinese text classification model in the present invention ( Figure 2 The red arrow in the figure is as follows:

[0161] 1. For the test set Perform text preprocessing (including sentence segmentation, stop word removal, and punctuation removal).

[0162] 2. Use RoBERTa to get the input text x i The word vector representation E(x i ), and get the embedding representation set of each text

[0163] 3. Multi-head attention representation: The embedding vector E(x i ) as input, and obtain long-term dependency information through the multi-head attention mechanism. Using Formula 7, we get the form H(x i )=[h1,h2,h3,…,h n ] is the feature vector of ].

[0164] 4. Text classification: Use the SoftMax function (normalized exponential function) to classify the text and use Formula 17 to obtain the classification label of the sample.

[0165] This implementation case applies the technical solution provided by the present invention to actual highly unbalanced Chinese text classification.

[0166] 1. Experimental Data

[0167] The corpus DataRT19-20 is used. It comes from real railway vehicle safety supervision data and is divided into 77 categories according to business needs. The training data set is the data for the whole year of 2019, and the test set is the data from January to April 2020. There are 27092 training data and 2712 test data. Table 1 gives the detailed statistics of the training data set and validation data set in DataRT19-20. It can be seen from the training data in the table that most of the 77 categories have thousands of data, while a few categories have only a few data, the data is highly unbalanced, and the data of a few categories is extremely scarce.

[0168] All experiments used PyTorch to build the model network and train the parameters. The hyperparameters were adjusted through 5-fold cross-validation, and the final model was trained on the best combination set.

[0169] Table 1 List of types and quantities in DataRT19-20

[0170]

[0171]

[0172] 2. Experimental results

[0173] (1) Comparison of text classification model effects

[0174] First, based on the same dataset DataRT19-20, the performance of the text classification model of the present invention is compared with other existing commonly used text classification models. Table 2 lists the comparison based on classification accuracy.

[0175] Table 2. Performance comparison of text classification models

[0176] Text Classification Model accuracy Text-CNN 0.65 Bert-base 0.705 Roberta-base 0.713 Ernie 0.715 Electra-base 0.702 Our system 0.788

[0177] Among them, Text-CNN is the most commonly used text classification model, which is built based on convolutional neural networks; Bert-base is a text classification model based on the Bert network proposed by Google; Roberta-base is a text classification model based on the Roberta network proposed by Facebook; Ernie is a text classification model based on the Ernie network proposed by Baidu; Electra-base is a text classification model based on the Electra network proposed by Stanford SAIL Laboratory; Our system is a text classification model of data enhancement and contrastive learning proposed in this invention.

[0178] The results in Table 2 show that the text classification model proposed in this invention can achieve the best overall performance and has significant differences compared with other systems.

[0179] (2) Performance Analysis of Data Augmentation

[0180] The following is a comparison of the performance of self-constrained stacked data enhancement. The baseline model of the present invention is Roberta-base, on which an MLM-based text enhancer and a perturbation-based text enhancer are added respectively, to obtain the models Roberta-base+MLM and Roberta-base+perturbation. Our system (w\o DW) uses the stacked data enhancement module proposed in the present invention, but does not use the contrastive learning module. The performance comparison is shown in Table 3.

[0181] Table 3 Performance comparison of data augmentation

[0182] Text Classification Model accuracy Roberta-base 0.713 Roberta-base+MLM 0.721 Roberta-base+perturbation 0.722 Our system(w\o DW) 0.756

[0183] From the results in Table 3, we can see that the performance of the models after using self-constrained stacking data enhancement is higher than the original model, which shows that this method can effectively enhance the data and improve the imbalance between categories.

[0184] (3) Performance Analysis of Double Weighted Contrastive Learning

[0185] The following compares the performance of the contrastive learning method based on double weighting. Comparing the three text classification models Oursystem(W1), Our system(W2) and Our system, it means that on the basis of the Our system(w\o DW) model, contrastive learning is adopted and weight functions W1, W2, W1 and W2 are added respectively.

[0186] Table 4 Performance comparison of representation fusion

[0187] Text Classification Model accuracy Our system(w\o DW) 0.756 <![CDATA[Our system(W1)]]> 0.770 <![CDATA[Our system(W2)]]> 0.768 Our system 0.788

[0188] From the results in Table 4, it can be seen that after using the contrastive learning method based on different weights, the model performance is improved compared with the performance of the original text classification model, indicating that this method can further improve data distribution, optimize data representation, and improve model classification performance on the basis of data enhancement.

[0189] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. An enhanced contrastive learning method for highly imbalanced text classification, characterized in that: The following steps are involved: S1, dividing the collected text data set into a training data set and a test data set, preprocessing the text data in the training data set and the test data set respectively to obtain the corresponding original training text and test text; S2, input the original training text and test text into the RoBERTa model respectively to obtain the corresponding text embedding representation; S3, constructing a text classification model based on data enhancement and contrastive learning, wherein the text classification model includes a text enhancement module, a text representation module, a contrastive learning module and a text classification module, using the original training text and its embedded representation to train the text classification model, and using the embedded representation of the test text to test the trained text classification model; S4. Apply the trained and tested text classification model to the classification of the text data to be processed, and output the classification result of the text data to be processed.

2. The enhanced contrastive learning method for highly imbalanced text classification according to claim 1, characterized in that: In step S1, the preprocessing operations include sentence segmentation, word segmentation, stop word removal and punctuation removal; The original training text is represented as D orig , Among them, x i is the i-th text in the training dataset, y i is the type label corresponding to the i-th text, and N1 is the number of texts in the training data set; In step S2, RoBERTa obtains the input text x i The word vector embedding representation E(x i ), and get the embedding representation set of each text 3. The enhanced contrastive learning method for highly imbalanced text classification according to claim 2, characterized in that: In step S3, the text enhancement module includes a text enhancer based on a masked language model and a text enhancer based on perturbations. The text enhancer based on the masked language model proposes a data enhancement algorithm for dynamically randomly selecting a mask. The text in is masked, and then the text is reconstructed by fine-tuning the RoBERTa model, outputting a set of text embedding representations of N2 is x i Corresponding enhanced text; Perturbation-based text augmentation uses adversarial training to augment text collections. Add adversarial perturbations and output a set of N2 enhanced data embedding representations yes Corresponding enhanced text; The text enhancement module designs a consistent self-constrained loss function L consis : Among them, L BCE is the loss function of the text classification module, R con To introduce the label consistency loss described by KL divergence, δ and ξ are the corresponding loss weight coefficients.

4. The enhanced contrastive learning method for highly imbalanced text classification according to claim 3, characterized in that: The data enhancement algorithm processing flow of the dynamic random selection mask is as follows: Obtain a sentence obtained through the word segmentation operation in preprocessing, where the sentence is represented as a set of multiple words; Randomly select words in the masked sentence at a preset ratio, and replace the masked words with randomly generated masks to obtain the masked sentence; Merge consecutive mask tokens to get the merged sentence.

5. The enhanced contrastive learning method for highly imbalanced text classification according to claim 3, characterized in that: In step S3, the text x i The word vector embedding representation E(x i ) and word embedding representation of enhanced text Constitute the total text representation set Input text representation module; The text representation module obtains long-distance dependency information through a multi-head attention mechanism and outputs the feature vector representation H(x) of the text, where H(x) = [h1,h2,h3,…,h j ,…,h n ].

6. The enhanced contrastive learning method for highly imbalanced text classification according to claim 5, characterized in that: In step S3, the contrastive learning module performs the following process: Construct positive and negative sample pairs for contrastive learning: Given a training batch with M samples, for each text d in the training batch i , construct positive sample pairs p for contrastive learning pos =(d i ,d j(i) ), that is, with the same label y i The sample pairs can be obtained by M1 is the number of positive sample pairs; for each text d in the training batch i , construct negative sample pairs p for contrastive learning neg =(d i ,d k ), that is, sample pairs with different labels, we can get a set of negative sample pairs M2 is the number of negative sample pairs; then the loss function of classic contrastive learning can be obtained: Among them, the temperature coefficient τ is a hyperparameter used to adjust the degree of penalty for the model to focus on difficult negative samples; Design a doubly weighted contrastive learning loss function L DWCL : W1, W2 are weight functions, l(d i ) is the loss of each sample in the training batch; 0<μ<1; The text representation module improves the feature space distribution by constructing positive and negative sample pairs and a double weighted contrast learning loss function training strategy, and outputs the optimized text vector representation H(x i ).

7. The enhanced contrastive learning method for highly imbalanced text classification according to claim 6, characterized in that: In step S3, the text classification module inputs the feature vector H(x) output by the text representation module and designs a cross entropy loss function L BCE , use the SoftMax function to classify the text and output the probability of text classification; The consistency self-constraint loss function L consis , double weighted contrastive learning loss function L DWCL And the cross entropy loss function L BCE Together they constitute the total loss function L of the text classification model based on data enhancement and contrastive learning: L=L BCE +αL consis +βL DWCL Among them, α and β are the corresponding loss weight coefficients; According to the total loss function L, the total loss of the model is calculated and derived, the error is back-propagated, the model parameters are optimized, and the model training is completed.

8. An enhanced contrastive learning device for highly imbalanced text classification, characterized in that: include: The data preprocessing module is used to divide the collected text data set into a training data set and a test data set, and preprocess the text data in the training data set and the test data set respectively to obtain the corresponding original training text and test text; The RoBERTa module is used to input the original training text and test text into the RoBERTa model to obtain the corresponding text embedding representation; A model construction module is used to construct a text classification model based on data enhancement and contrastive learning, wherein the text classification model includes a text enhancement module, a text representation module, a contrastive learning module and a text classification module. The text classification model is trained using the original training text and its embedded representation, and the trained text classification model is tested using the embedded representation of the test text. The classification processing module is used to apply the text classification model that has completed training and testing to the classification processing of the text data to be processed, and output the classification results of the text data to be processed.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of any method described in claims 1-7 are implemented.

Images