A fine-grained text classification method based on supervised contrastive learning
By introducing joint training with supervised contrast learning and cross-entropy loss in fine-grained text classification, combined with random word replacement data augmentation, the semantic similarity problem caused by category hierarchy relationships is solved, and the effective classification of fine-grained texts is achieved.
Patent Information
- Application Number
- CN202210430095.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-04-22
AI Technical Summary
The prior art is difficult to distinguish between class relationships with subtle differences in fine-grained text classification, especially due to the semantic similarity problem caused by the hierarchical relationship between classes. The cross-entropy loss function cannot effectively distinguish between semantic differences in different classes.
Using a method based on supervised contrast learning, by constructing a hierarchical classification system, defining positive and negative examples, and using joint training of cross entropy loss and contrast loss, combined with data augmentation technology of random word replacement, ensure the semantic distance of the same category of samples. In negative examples.
Effectively distinguishing semantic differences in fine-grained texts improves the ability to identify category-level relationships, ensures that the semantic distances of the same category samples are close, and improves the accuracy of fine-grained text classification.
Smart Images

Figure CN114757289B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a text classification method, in particular to a fine-grained text classification method based on supervised contrastive learning. Background Art
[0002] Fine-grained text classification needs to distinguish between slightly different inter-class relationships. For example, in fine-grained sentiment classification, it is necessary to distinguish not only the sentiment category but also the sentiment intensity. Therefore, it is necessary to distinguish not only the two categories of praise and derogatory, but also the degree of praise and derogatory. Therefore, each category is further divided into general expression and extreme description of sentiment. For example, praise can be further divided into general praise and extreme praise.
[0003] This paper aims to solve the problem of fine-grained text classification caused by the hierarchical relationship between categories. Because categories are hierarchical tree structures, the bottom leaf categories often have the same parent category. For example, the category "hostile-verbal-rejection" and the category "hostile-verbal-opposition" have the same parent category "hostile-verbal", and only the leaf categories are different. Because of the same parent category, the texts of different categories are very similar in semantics. To solve this problem, we should not only distinguish the categories to which the samples belong, but also distinguish the semantic differences between the samples.
[0004] For the problem of general text classification, the existing solution is to obtain the vector representation of the text through the pre-trained language model BERT, and then use the cross-entropy loss function for fine-tuning. However, the cross-entropy loss function is not suitable for fine-grained classification, because the exclusivity of the cross-entropy loss will treat each category equally. For example, there is no difference in the loss function when a sample that was originally generally positive is misclassified as extremely positive or extremely negative, but in fact the semantic distance between the generally positive sample and the extremely positive sample is closer to the extremely negative sample. Contrastive learning is precisely to solve this problem. It can ensure that the distance between the current sample and the positive example is closer to the negative example. Therefore, we introduced supervised contrast loss for joint training, and reasonably designed positive and negative examples for the current sample. The contrast loss can ensure that the distance between the current sample and the positive example is closer to the distance between the negative example. Summary of the invention
[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a fine-grained text classification method based on supervised contrastive learning in view of the shortcomings of the prior art.
[0006] In order to solve the above technical problems, the present invention discloses a fine-grained text classification method based on supervised contrastive learning, comprising the following steps:
[0007] Step 1: Build a text classification model and define a hierarchical classification system to describe categories in a fine-grained manner;
[0008] Step 2: Select samples and reasonably construct positive and negative examples for each sample to perform data augmentation;
[0009] Step 3: Jointly train the text classification model based on cross entropy loss and contrast loss to achieve fine-grained text classification.
[0010] In the present invention, step 1 comprises:
[0011] Define a hierarchical classification system to describe the hierarchical relationship between categories. Labels at different levels are separated by the symbol -. By defining this label, we hope to provide document-level classification functions for news in the fields of politics, military affairs, diplomacy, etc.
[0012] In the present invention, step 2 comprises:
[0013] A batch of samples is selected from the text data set as training samples, that is, the number of samples in the batch is defined as K, the positive sample set in the batch is P, the negative sample set is N, and the sample x in the batch is defined as i and its label y i For the set {x i ,y i} i∈I , where set I = {1,…,K};
[0014] Step 2-1, construct a training data set;
[0015] Step 2-2, construct positive and negative examples;
[0016] Step 2-3, data augmentation based on random word replacement.
[0017] In the present invention, step 2-1 comprises:
[0018] For the selected samples, construct training corpus; the training corpus includes article titles, article contents and hierarchical classification labels of the articles;
[0019] Preprocessing the training corpus; the preprocessing includes: converting traditional Chinese characters into simplified Chinese characters, and converting full-width numbers and full-width letters into half-width numbers and half-width letters;
[0020] The article title and the body are concatenated with a period to determine whether the length exceeds a preset length threshold; if not, the concatenated result is used as the article; if the length exceeds the preset length threshold, the article is truncated and the truncated content is used as the article;
[0021] Perform the above operations on each sample to obtain the training data set.
[0022] In the present invention, the positive and negative example construction method described in step 2-2 includes:
[0023] For each training data set obtained in step 2-1, that is, each training sample x i , where the positive examples are defined as samples with the same category label and their augmented samples, and the negative examples are defined as samples with different category labels and their augmented samples:
[0024] P={p:p∈I,y p =y i ∧p≠i}
[0025] N={p:p∈I,y p ≠y i}
[0026] Where I represents the set of all sample subscripts, p is an element in set I, and y p Its corresponding label, y i Represents sample x i The category of P represents the sample x i The positive sample set, N represents its negative sample set;
[0027] The data augmentation method based on random word replacement in step 2-3 includes:
[0028] The data in the training data set constructed by positive and negative examples in step 2-2 is segmented using the Jieba word segmenter, and the following four replacement methods are randomly selected:
[0029] Replacement method 1, synonym replacement: randomly select n words and replace each selected word with a synonym;
[0030] Replacement method 2, random insertion: randomly select a word in the sentence, find its synonym, insert the synonym into a random position in the sentence, and repeat this process n times;
[0031] Replacement method 3, random replacement: randomly select two words in the sentence, and then exchange their positions with each other, and repeat this process n times;
[0032] Replacement method 4, random deletion: Delete each word in the sentence with probability p, and the total number of deleted words is m;
[0033] m=p(del)l
[0034] Where l represents the length of the sentence, and p(del) represents the probability of deleting a word at each position;
[0035] After replacement, the training data set is obtained after data augmentation.
[0036] In the present invention, step 3 comprises:
[0037] Step 3-1, obtain the semantic vector through BERT encoding;
[0038] Step 3-2, calculate the contrast loss to bring the samples of the same category closer;
[0039] Step 3-3, calculate the cross entropy loss of text classification;
[0040] Step 3-4, construct a joint loss function and jointly train the text classification model to achieve fine-grained text classification.
[0041] In the present invention, step 3-1 obtains a semantic vector by BERT encoding (BERT is a text encoding method, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding), and the method includes:
[0042] For the training samples after data augmentation, two special tags [CLS] and [EOS] are added to the first position of the bert segmentation number, marked as:
[0043] x i =[CLS],t1,t2,…,t L ,[EOS]
[0044] Where L is the document length, for the sample sequence x i The vector after bert feature extraction is marked as h i ;
[0045] In the present invention, the calculation of contrast loss in step 3-2 shortens the distance L between samples of the same category. cl , methods include:
[0046]
[0047] Where i is a natural number ranging from 1 to K, x i represents each sample in the augmented batch; τ is a temperature parameter with a value between 0 and 1, h p Represents the positive sample x of the current sample p The vector representation after BERT encoding, h i is the current sample x i The normalized representation of the vector of the BERT encoding number, k is the other training samples in the set after removing the current training sample i, h k is its BERT semantic encoding.
[0048] In the present invention, the method for calculating the text classification cross entropy loss in step 3-3 includes:
[0049] For the augmented sample x i Calculate the cross entropy loss L CE , methods include:
[0050]
[0051] Among them, C represents the number of categories, y i,c is the true label of the sample, is the model output, representing the sample x i The probability of being in class c.
[0052] In the present invention, the method for constructing the joint loss function L in step 3-4 includes:
[0053] L=(1-λ)L CE +λL cl
[0054] Among them, λ is a hyperparameter that controls the weight of the two loss functions.
[0055] Beneficial effects:
[0056] Aiming at the actual needs of fine-grained text classification, a hierarchical classification system is defined; in order to distinguish fine-grained text categories, a loss function based on contrastive learning is introduced; in order to construct positive examples of samples, a data augmentation method based on random replacement is proposed; a fine-grained text classification method based on the combination of contrast loss and cross entropy loss is proposed, and the idea of contrastive learning is introduced to solve the problem of fine-grained text classification, ensuring a close semantic distance between samples of the same category. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.
[0058] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0059] The present invention first defines the text type of interest and defines a hierarchical classification system; performs data augmentation on samples using the idea of random replacement; reasonably defines positive and negative examples for given samples, wherein samples of the same category and their augmented samples are used as positive examples, and samples of other categories are used as negative examples; by reasonably defining positive and negative examples and using the idea of contrastive learning, it is ensured that the semantic distance of samples of the same category is closer than that of samples of different categories, thereby solving the problem of fine-grained classification caused by the hierarchical relationship of categories.
[0060] like Figure 1 As shown, firstly, the training sample x a Construct positive example x p and negative example xn , and then each of them is replaced by a random word to form 5 sample pairs, where x a ',x p ,x p 'Constitute x a A positive example, x n and x n 'Constitute x a Negative examples. Use these 5 sample pairs for contrast loss training. a The text is classified by a classifier. The present invention provides a fine-grained text classification method based on supervised contrastive learning, which specifically includes the following steps:
[0061] Step 1 mainly defines a hierarchical classification system to describe the hierarchical relationship between categories, such as "collaboration-verbal-expression of willingness-substantial cooperation". Different levels of labels are separated by "-". By defining this label, we hope to provide document-level classification functions for news in the fields of politics, military affairs, diplomacy, etc.
[0062] Step 2 includes:
[0063] Define the number of samples in a batch as K, the set of positive samples in the batch as P, the set of negative samples as N, and define the samples and their labels in the batch as the set {x i ,y i} i∈I , where I = {1,…,K}
[0064] Step 2-1: Construct a training data set, including the following steps:
[0065] Collecting training corpus, wherein the training corpus includes article titles, article contents, and hierarchical classification labels of articles;
[0066] Preprocessing the training corpus includes: converting traditional Chinese characters into simplified Chinese characters, and converting full-width numbers and full-width letters into half-width numbers and half-width letters;
[0067] The article title and the body are concatenated with a period to determine whether the length exceeds a preset length threshold; if not, the concatenated result is used as the article; if the length exceeds the preset length threshold, the article is truncated and the truncated content is used as the article;
[0068] Step 2-2: Data augmentation based on random word replacement
[0069] First, use Jieba to segment the original sample and randomly select the following four replacement methods:
[0070] Synonym replacement: randomly select n words and replace each selected word with a synonym;
[0071] Random insertion: randomly select a word in the sentence, then find its synonym, and then insert the synonym into a random position in the sentence, and repeat this process n times;
[0072] Random replacement: randomly select two words in the sentence and then swap their positions. This process is repeated n times.
[0073] Random deletion: Delete each word in the sentence with probability p.
[0074] Where l represents the length of the sentence, and p represents the probability of deleting a word at each position.
[0075] n=pl
[0076] Step 2-3: Construction of positive and negative examples:
[0077] For each sample x i ,The positive examples are defined as samples with the same category label and their augmented samples, and the negative examples are defined as samples with different category labels and their augmented samples.
[0078] P={p:p∈I,y p =y i ∧p≠i}
[0079] N={p:p∈I,y p ≠y i}
[0080] Step 3 includes:
[0081] Step 3-1: BERT encoding vector
[0082] For the original document, two special tags [CLS] and [EOS] are added to the first position of the bert segmentation number, marked as
[0083] x i =[CLS],t1,t2,…,t L ,[EOS]
[0084] Where L is the document length, for the sequence x i The vector after bert feature extraction is marked as h i
[0085] Step 3-2: Contrastive loss brings samples of the same category closer together:
[0086]
[0087] Where i represents each sample x in the batch i, the batch size was originally K. Because each sample was augmented, the batch size became 2K. τ is a temperature parameter with a value of (0,1). For negative samples that are difficult to distinguish, the network originally has a large output value. If the temperature parameter is taken as a large value, the weight of the negative sample will be reduced. On the contrary, the smaller the value of the parameter is, the weight of the negative sample that is difficult to distinguish will be increased accordingly, thereby generating more negative samples that are difficult to distinguish; h i is the current sample x i The normalized representation of the vector of the BERT encoding number.
[0088] Step 3-3: Cross Entropy Loss for Text Classification
[0089] For the sample x i Calculate the cross entropy loss,
[0090]
[0091] Where C represents the number of categories, y i,c is the true label of the sample, is the probability that the model output sample i belongs to category c;
[0092] Step 3-4: Joint loss function
[0093] L=(1-λ)L CE +λL cl
[0094] Among them, λ is a hyperparameter that controls the weight of the two loss functions.
[0095] Example:
[0096] The following two samples belong to two different categories, but because there is overlap between the two categories, they have the same semantic expression. Ordinary text classification has a poor expression effect on texts with similar semantics.
[0097] Sample 1: According to Yonhap News Agency, XXX, who is visiting the United States, said in an interview with Korean media in Washington, "I still can't understand (the protest of XX and others)" and refused to apologize to Japan.
[0098] Sample 1 Label: Hostile-Verbal-Rejection
[0099] Sample 2: XXX said that Iran rejected XXX's request to visit Tehran and said that one of the mediators was XXXX
[0100] Sample 2 Label: Hostile-Verbal-Refuse to Communicate"
[0101] According to step 2, construct positive and negative examples of the sample. The positive sample is constructed by randomly selecting one from the same category. Taking sample 1 as an example, the positive example is "hostile-verbal-rejection". A sample is randomly selected, and the negative example is selected from any other category except "hostile-verbal-rejection". Then perform data augmentation on the positive and negative examples and themselves to form augmented samples. According to step 3, perform joint training of cross entropy loss and contrastive learning loss.
[0102] The present invention provides an idea and method for fine-grained text classification based on supervised contrastive learning. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.
Claims
1. A fine-grained text classification method based on supervised contrastive learning, characterized in that: The following steps are involved: Step 1: Build a text classification model and define a hierarchical classification system to describe categories in a fine-grained manner; Step 2: Select samples and reasonably construct positive and negative examples for each sample to perform data augmentation; Step 3: Jointly train the text classification model based on cross entropy loss and contrast loss to achieve fine-grained text classification; Wherein, step 3 comprises: Step 3-1, obtain the semantic vector through BERT encoding; Step 3-2, calculate the contrast loss to bring the samples of the same category closer; Step 3-3, calculate the cross entropy loss of text classification; Step 3-4, construct a joint loss function, jointly train the text classification model, and achieve fine-grained text classification; Step 3-1 obtains the semantic vector by BERT encoding, and the method includes: For the training samples after data augmentation, two special tags [CLS] and [EOS] are added to the first position of the bert segmentation number, marked as: x i =[CLS],t1,t2,…,t L ,[EOS] Where L is the document length, for the sample sequence x i The vector after bert feature extraction is marked as h i ; The calculation of contrast loss in step 3-2 brings the distance L of samples of the same category closer cl , methods include: Where i is a natural number ranging from 1 to K, x i represents each sample in the augmented batch; τ is a temperature parameter with a value between 0 and 1, h p Represents the positive sample x of the current sample p The vector representation after BERT encoding, h i is the current sample x i The normalized representation of the vector of the BERT encoding number, k is the other training samples in the set after removing the current training sample i, h k is its BERT semantic encoding; The method for calculating the text classification cross entropy loss in step 3-3 includes: For the augmented sample x i Calculate the cross entropy loss L CE , methods include: Among them, C represents the number of categories, y i,c is the true label of the sample, is the model output, representing sample x i The probability of being in category c; The method for constructing the joint loss function L in step 3-4 includes: L=(1-λ)L CE +λL cl Among them, λ is a hyperparameter that controls the weight of the two loss functions.
2. According to claim 1, a fine-grained text classification method based on supervised contrastive learning is characterized in that: Step 1 includes: Define a hierarchical classification system to characterize the hierarchical relationship between categories. Labels at different levels are separated by the symbol -. By defining this label, document-level classification function is provided.
3. According to claim 2, a fine-grained text classification method based on supervised contrastive learning is characterized in that: Step 2 includes: A batch of samples is selected from the text data set as training samples, that is, the number of samples in the batch is defined as K, the positive sample set in the batch is P, the negative sample set is N, and the sample x in the batch is defined as i and its label y i For the set {x i ,y i } i∈I , where set I = {1,…,K}; Step 2-1, construct a training data set; Step 2-2, construct positive and negative examples; Step 2-3, data augmentation based on random word replacement.
4. According to claim 3, a fine-grained text classification method based on supervised contrastive learning is characterized in that: Step 2-1 includes: For the selected samples, construct training corpus; the training corpus includes article titles, article contents and hierarchical classification labels of the articles; Preprocessing the training corpus; the preprocessing includes: converting traditional Chinese characters into simplified Chinese characters, and converting full-width numbers and full-width letters into half-width numbers and half-width letters; The title and the body of the article are concatenated with a period to determine whether the length exceeds a preset length threshold; if not, the concatenated result is used as the article; if the length exceeds the preset length threshold, the article is truncated and the truncated content is used as the article; Perform the above operations on each sample to obtain the training data set.
5. According to claim 4, a fine-grained text classification method based on supervised contrastive learning is characterized in that: The positive and negative example construction method described in step 2-2 includes: For each training data set obtained in step 2-1, that is, each training sample x i , where the positive examples are defined as samples with the same category label and their augmented samples, and the negative examples are defined as samples with different category labels and their augmented samples: P={p:p∈I,y p =and i ∧p≠i} N={p:p∈I,y p ≠y i } Where I represents the set of all sample subscripts, p is an element in set I, and y p Its corresponding label, y i Represents sample x i The category of P represents the sample x i The positive sample set, N represents its negative sample set; The data augmentation method based on random word replacement in step 2-3 includes: The data in the training data set constructed by positive and negative examples in step 2-2 is segmented using the Jieba word segmenter, and the following four replacement methods are randomly selected: Replacement method 1, synonym replacement: randomly select n words and replace each selected word with a synonym; Replacement method 2, random insertion: randomly select a word in the sentence, find its synonym, insert the synonym into a random position in the sentence, and repeat this process n times; Replacement method 3, random replacement: randomly select two words in the sentence, and then exchange their positions with each other, and repeat this process n times; Replacement method 4, random deletion: Delete each word in the sentence with probability p, and the total number of deleted words is recorded as m; m=p(del)l Where l represents the length of the sentence, and p(del) represents the probability of deleting a word at each position; After replacement, the training data set is obtained after data augmentation.
