An imbalanced text classification method introducing keyword features

Abstract

The present invention provides an unbalanced text classification method that introduces keyword features. First, a hierarchical classification system is defined for the field of military news, including 32 leaf categories; keywords of each category are extracted using normalized point mutual information and improved information gain; keyword features and neural network semantic features are integrated for training. Through the above steps, the present invention introduces class label distribution as a priori information into the text classification model for the problem of unbalanced text classification, and the training process simultaneously uses text content and category keyword information; uses normalized point mutual information and improved information gain as statistics for category keyword selection to select category keywords; uses keyword information and text information to jointly train the text classification model, effectively solving the category imbalance problem in text classification.

Description

Technical Field

[0001] The invention relates to a text classification method, in particular to an imbalanced text classification method introducing keyword features. Background Art

[0002] Category imbalance refers to the long tail phenomenon in the corpus, where the amount of data in each category varies greatly. Existing text classification models are essentially optimizations of loss functions, minimizing a certain loss function. In imbalanced text classification, because the sample sizes of each category are different, the loss function of samples that appear on a large scale will account for a high proportion, which will eventually cause the model to favor categories that appear on a large scale.

[0003] For the problem of general text classification, the existing solution is to obtain the vector representation of the text through the pre-trained language model BERT, and then fine-tune it using the cross entropy loss function. For the problem of class imbalance, there are mainly two ways to improve it: data resampling and model improvement by modifying the class weights in the loss function. However, both methods cannot solve the underfitting problem of sparse categories. Suppose a sparse category has only one sample, but the sample contains more words, and the combination of these words only appears in this category. Neither resampling nor class weighting can solve the diversity problem of the washing sample.

[0004] The category distribution in the training corpus is an important prior information, which is crucial for the classification model, because this information will prompt the model the probability of a certain category appearing, which is irrelevant to the text content. Therefore, how to introduce category distribution information into the model is the key problem to be solved in this paper. How to represent category labels? This paper uses normalized point mutual information (NPMI) and improved information gain (IG+) to filter category keywords as the prior representation of class labels, and introduces this information into the text classification model during model training. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide an unbalanced text classification method by introducing keyword features in view of the deficiencies in the prior art.

[0006] In order to solve the above technical problems, the present invention discloses an unbalanced text classification method introducing keyword features, comprising the following steps:

[0007] Step 1: Build a text classification model, define a hierarchical classification system, build category labels, and build a training data set;

[0008] Step 2, extract keywords of each category in the text to obtain keyword information;

[0009] Step 3: Integrate keyword information and text semantic information for model training.

[0010] In the present invention, step 1 comprises:

[0011] Step 1-1, define a hierarchical classification system, characterize the hierarchical relationship between categories, separate labels at different levels by symbols, and perform article-level classification;

[0012] Step 1-2, building a training data set, includes the following steps:

[0013] Collecting training corpus; the training corpus includes article titles, article contents and hierarchical classification labels of articles;

[0014] Preprocessing the training corpus; the preprocessing includes: converting traditional Chinese characters into simplified Chinese characters, and converting full-width numbers and full-width letters into half-width numbers and half-width letters;

[0015] The article title and body are concatenated with a period to determine whether the length exceeds a preset length threshold; if not, the concatenated result is used as the article; if it exceeds the preset length threshold, the article is truncated and the truncated content is used as the article.

[0016] In the present invention, step 2 comprises:

[0017] Step 2-1: Select category keywords based on normalized point mutual information;

[0018] Step 2-2: Select category keywords based on improved information gain IG+;

[0019] Step 2-3: Select a set of keywords for each category.

[0020] In the present invention, the method for selecting category keywords based on normalized point mutual information in step 2-1 includes:

[0021] The text in the training data set constructed in step 1-2 is segmented to form a word sequence, and the occurrence of each word and text category is counted. The normalized point mutual information npmi(w,y) between word w and category y is calculated as follows:

[0022]

[0023] Among them, p(w,y) represents the number of co-occurrences of word w and category y divided by the total number of training texts; p(w) and p(y) represent the total number of word and category occurrences divided by the total number of training texts, respectively.

[0024] In the present invention, the method for selecting category keywords based on improved information gain described in step 2-2 includes:

[0025] Information gain considers this measurement from the perspective of information entropy, analyzing the contribution of a feature item to the information amount of the entire system. Its disadvantage is that it does not analyze the distribution of a feature within a category and the classification of a feature in all categories. Therefore, we introduce the intra-class dispersion degree DI and the inter-class concentration degree CI, and use the improved information gain IG+ method to calculate the IG+ score IG+(w,c) of the keyword of the category. The method includes:

[0026] IG+(w,c)=(DI(w,c)-CI(w,c))IG(w,c)

[0027] Among them, DI represents the degree of dispersion within the class, which means the ratio of the number of occurrences of word w in the text of category c to the total number of occurrences of the text of this category. The larger the value, the wider the distribution of the word in this type of text, and the more representative the word is of this category. The calculation method is:

[0028]

[0029] Where df(w,c) represents the number of texts in category c that contain word w, and df(c) represents the number of texts in category c;

[0030]

[0031] Among them, CI(w,c) represents the degree of inter-class concentration, that is, the ratio of the number of categories in which word w appears to the total number of categories. The smaller the ratio, the fewer categories the word appears in, and the more representative it is of these categories. Among them, |C| represents the total number of categories, and cf(w) represents the number of categories in which word w appears.

[0032] The information gain IG(w) of word w is used to measure the degree of uncertainty of text classification caused by the presence or absence of word w. The calculation method includes:

[0033]

[0034] Where n represents the total number of text categories, i represents the text traversed in each category; p(w) represents the probability of word w appearing in the text; p(C i |w) represents the posterior probability for the word, that is, the text containing word w belongs to analogy C i probability; Indicates the probability that the text does not contain word w; Indicates that the text does not contain word w and belongs to category C i The posterior probability of .

[0035] In the present invention, the method for selecting a keyword set for each category in step 2-3 includes:

[0036] Traverse each category, calculate the npmi and IG+ scores of each word, and arrange them in descending order. Select the top 10 words with the highest scores as the word representations of the category, and get 20 keyword representations for each category.

[0037] In the present invention, in the model training process of integrating keyword information and text information in step 3, for each batch:

[0038] Step 3-1, calculate the text cross entropy loss function within a single batch;

[0039] Step 3-2, randomly select N C categories, traverse each category, sample K keywords for each category, and concatenate the keywords to form the keyword text of the category;

[0040] Step 3-3, calculate the cross entropy loss for the keyword text;

[0041] Step 3-4, calculate the joint loss.

[0042] In the present invention, the calculation of the cross entropy loss function of text classification in a single batch in step 3-1 includes:

[0043] After bert segmentation, special tags [CLS] and [EOS] are added to the beginning and end of the original text, recorded as:

[0044] x doc =[CLS], t 1 , t 2 ,...t j , ..., t L , [EOS]

[0045] Where L is the length of the text, t L Represents the word unit after bert segmentation. For the sequence x doc , after bert feature extraction, calculate the cross entropy loss L doc , methods include:

[0046]

[0047] Among them, C represents the number of categories, y j，c is the true label of the sample, is the output sample x of the text classification model j The probability of belonging to category c, j is the sample number;

[0048] In the present invention, the random selection N in step 3-2 C categories, the number of categories N c The calculation methods include:

[0049] N c =max(batch_size, C)

[0050] Among them, C represents the total number of categories, batch_size represents the number of samples in this batch; for the keyword text x after keyword selection for the current category c, keyword It is expressed as:

[0051] x keyword =W 1 , w 2 ... K

[0052] Among them, w K It is the word representing category c obtained after feature selection in steps 2-3.

[0053] In the present invention, the method for calculating the cross entropy loss for the keyword text in step 3-3 includes:

[0054] For keyword text x keyword According to the method described in step 3-1, after BERT segmentation, its semantic vector representation h is obtained. keyword , and calculate the cross entropy loss L after the keyword according to step 3-1 keyword ;

[0055] Step 3-4 calculates the joint loss based on the text and keywords. The calculation method includes:

[0056] Loss = L doc +αL keyword

[0057] Among them, L doc represents the cross entropy loss of the text, L keyword Represents the cross entropy loss of keywords, α is a hyperparameter with a value between 0 and 1.

[0058] Beneficial effects:

[0059] To address the problem of imbalanced text classification, the class label distribution is introduced into the text classification model as a priori information. The training process simultaneously utilizes text content and category keyword information. Normalized point mutual information and improved information gain are used as statistics for category keyword selection to perform category keyword selection. Keyword information and text information are used to jointly train the text classification model to effectively solve the problem of category imbalance in text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0061] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0062] like Figure 1 As shown, an imbalanced text classification method introducing keyword features includes the following steps:

[0063] Step 1: Build a text classification model, define a hierarchical classification system, build category labels, and build a training data set;

[0064] Step 1-1, define a hierarchical classification system, characterize the hierarchical relationship between categories, separate labels at different levels by symbols, and perform article-level classification;

[0065] Step 1-2, building a training data set, includes the following steps:

[0066] Collecting training corpus; the training corpus includes article titles, article contents and hierarchical classification labels of articles;

[0067] Preprocessing the training corpus; the preprocessing includes: converting traditional Chinese characters into simplified Chinese characters, and converting full-width numbers and full-width letters into half-width numbers and half-width letters;

[0068] The article title and body are concatenated with a period to determine whether the length exceeds a preset length threshold; if not, the concatenated result is used as the article; if it exceeds the preset length threshold, the article is truncated and the truncated content is used as the article.

[0069] Step 2, extract keywords of each category in the text to obtain keyword information;

[0070] Step 2-1: Select category keywords based on normalized point mutual information;

[0071] The text in the training data set constructed in step 1-2 is segmented to form a word sequence, and the occurrence of each word and text category is counted. The normalized point mutual information npmi(w,y) between word w and category y is calculated as follows:

[0072]

[0073] Among them, p(w,y) represents the number of co-occurrences of word w and category y divided by the total number of training texts; p(w) and p(y) represent the total number of word and category occurrences divided by the total number of training texts, respectively.

[0074] Step 2-2: Select category keywords based on improved information gain IG+;

[0075] Information gain considers this measurement from the perspective of information entropy, analyzing the contribution of a feature item to the information amount of the entire system. Its disadvantage is that it does not analyze the distribution of a feature within a category and the classification of a feature in all categories. Therefore, we introduce the intra-class dispersion degree DI and the inter-class concentration degree CI, and use the improved information gain IG+ method to calculate the IG+ score IG+(w,c) of the keyword of the category. The method includes:

[0076] IG+(w,c)=(DI(w,c)-CI(w,c))IG(w,c)

[0077] Among them, DI represents the degree of dispersion within the class, which means the ratio of the number of occurrences of word w in the text of category c to the total number of occurrences of the text of this category. The larger the value, the wider the distribution of the word in this type of text, and the more representative the word is of this category. The calculation method is:

[0078]

[0079] Where df(w,c) represents the number of texts in category c that contain word w, and df(c) represents the number of texts in category c;

[0080]

[0081] Among them, CI(w,c) represents the degree of inter-class concentration, that is, the ratio of the number of categories in which word w appears to the total number of categories. The smaller the ratio, the fewer categories the word appears in, and the more representative it is of these categories. Among them, |C| represents the total number of categories, and cf(w) represents the number of categories in which word w appears.

[0082] The information gain IG(w) of word w is used to measure the degree of uncertainty of text classification caused by the presence or absence of word w. The calculation method includes:

[0083]

[0084] Where n represents the total number of text categories, i represents the text traversed in each category; p(w) represents the probability of word w appearing in the text; p(C i |w) represents the posterior probability for the word, that is, the text containing word w belongs to analogy C i probability; Indicates the probability that the text does not contain word w; Indicates that the text does not contain word w and belongs to category C i The posterior probability of .

[0085] Step 2-3: Select a set of keywords for each category.

[0086] Traverse each category, calculate the npmi and IG+ scores of each word, and arrange them in descending order. Select the top 10 words with the highest scores as the word representations of the category, and get 20 keyword representations for each category.

[0087] Step 3: Integrate keyword information and text semantic information for model training.

[0088] Step 3-1, calculate the text cross entropy loss function within a single batch;

[0089] After bert segmentation, special tags [CLS] and [EOS] are added to the beginning and end of the original text, recorded as:

[0090] x doc =[CLS], t 1 , t 2 ,...t j , ..., t L , [EOS]

[0091] Where L is the length of the text, t L Represents the word unit after bert segmentation. For the sequence x doc , after bert feature extraction, calculate the cross entropy loss L doc , methods include:

[0092]

[0093] Among them, C represents the number of categories, y j，c is the true label of the sample, is the output sample x of the text classification model j The probability of belonging to category c, j is the sample number;

[0094] Step 3-2, randomly select N C categories, traverse each category, sample K keywords for each category, and concatenate the keywords to form the keyword text of the category;

[0095] The calculation method of the number of categories Nc includes:

[0096] N c =max(batch_size, C)

[0097] Among them, C represents the total number of categories, batch_size represents the number of samples in this batch; for the keyword text x after keyword selection for the current category c, keyword It is expressed as:

[0098] x keyword =w 1 , w2...w K

[0099] Among them, w K It is the word representing category c obtained after feature selection in steps 2-3.

[0100] Step 3-3, calculate the cross entropy loss for the keyword text;

[0101] For calculating the cross entropy loss for keyword text, the methods include:

[0102] For keyword text x keyword According to the method described in step 3-1, after BERT segmentation, its semantic vector representation h is obtained. keyword , and calculate the cross entropy loss L after the keyword according to step 3-1 keyword ;

[0103] Step 3-4, calculate the joint loss based on the text and keywords, the calculation method includes:

[0104] Loss = L doc +αL keyword

[0105] Among them, L doc represents the cross entropy loss of the text, L keyword Represents the cross entropy loss of keywords, α is a hyperparameter with a value between 0 and 1.

[0106] Example:

[0107] The present invention provides an unbalanced text classification method that introduces keyword features. First, a hierarchical classification system is defined for the military news field, including 32 leaf categories; the keywords of each category are extracted using normalized point mutual information and improved information gain; and the keyword features and neural network semantic features are integrated for training. Through the above steps, the present invention can effectively solve the text classification problem under the condition of unbalanced categories. Figure 1 As shown, the specific steps include:

[0108] Step 1 includes:

[0109] Step 1-1: Define a hierarchical classification system to describe the hierarchical relationship between categories, such as "collaboration-verbal-expression of willingness-substantial cooperation". Different levels of labels are separated by "-". By defining this label, we hope to provide text-level classification functions for news in the fields of politics, military affairs, diplomacy, etc.

[0110] Step: 1-2: Construct a training data set, including the following steps:

[0111] Collecting training corpus, wherein the training corpus includes article titles, article contents, and hierarchical classification labels of articles;

[0112] Preprocessing the training corpus includes: converting traditional Chinese characters into simplified Chinese characters, and converting full-width numbers and full-width letters into half-width numbers and half-width letters;

[0113] The article title and the body are concatenated with a period to determine whether the length exceeds a preset length threshold; if not, the concatenated result is used as the article; if the length exceeds the preset length threshold, the article is truncated and the truncated content is used as the article;

[0114] Step 2-1: Category keyword selection based on normalized point mutual information (npmi)

[0115] First, we use the word segmentation tool Jieba to segment the text into word sequences and count the occurrence of each word and text category. The NPM calculation formula for word w and category y is as follows:

[0116]

[0117] Where p(w,y) represents the number of co-occurrences of word w and category y divided by the total number of training texts; p(w) and p(y) represent the total number of word or category occurrences divided by the total number of training texts, respectively.

[0118] Step 2-2: Category keyword selection based on improved information gain (IG+)

[0119] Information gain considers this metric from the perspective of information entropy, analyzing the contribution of a feature item to the information content of the entire system. Its disadvantage is that it does not analyze the distribution of a feature within a category and the classification of a feature in all categories. Therefore, we introduce the intra-class dispersion (DI) and inter-class concentration (CI) and propose an improved information gain (IG+) method to calculate the keywords of a category.

[0120] IG+(w,c)=(DI(w,c)-CI(w,c))IG(w,c)

[0121] DI represents the degree of dispersion within the class, which means the ratio of the number of occurrences of word w in the text of category c to the total number of occurrences of the text of this category. The larger the value, the wider the distribution of the word in this type of text, and the more representative the word is of this category. df(w,c) represents the number of texts in category c that contain word w, and df(c) represents the number of texts in category c.

[0122]

[0123] CI(w,c) represents the degree of inter-class concentration, which is the ratio of the number of categories in which word w appears to the total number of categories. The smaller the ratio, the fewer categories the word appears in, and the more representative it is of these categories. |C| represents the total number of categories. c f(w) represents the number of categories in which word w appears:

[0124]

[0125] Where IG(w, c) represents the information gain of word w for category c, C is classified into two categories, belonging to category c and not belonging to category c, where p(C i ) indicates category C i The probability of occurrence, p(w) represents the probability of occurrence of word w, p(C i |w) indicates that category C occurs when word w appears i Probability of occurrence; represents the probability that word w does not appear, Indicates category C when word w does not appear i Probability

[0126]

[0127] Step 2-3: Select a set of keywords for each category

[0128] Traverse each category, calculate the npmi and IG+ scores of each word, and arrange them in descending order. Select the top 10 words with the highest scores as the word representations of the category, and get 20 keyword representations for each category.

[0129] Step 3 is the model training process that integrates keyword information and text information. For each batch, the following steps are performed:

[0130] Step 3-1: Calculate the text of a single batch

[0131] For the original text, two special tags [CLS] and [EOS] are added to the first word after bert segmentation, marked as

[0132] x doc =[CLS], t 1 , t 2 , ..., t L , [EOS]

[0133] Where L is the length of the text, for the sequence x doc The vector after bert feature extraction is marked as h doc , calculate the cross entropy loss

[0134]

[0135] Where C represents the number of categories, y i，c is the true label of the sample, is the probability that the model output sample i belongs to category c;

[0136] Step 3-2: Randomly select N C categories, and traverse each category, sample K keywords for each category, and concatenate the keywords to form the keyword text of the category; where Nc takes the value of max(batch_size, C), C represents the total number of categories, and the text after keyword selection for the current category c is represented as x keyword =w 1 , w 2 ... K

[0137] Step 3-3: Calculate the cross entropy loss for the keyword text:

[0138] For keyword text x keyword According to step 3-1, the semantic vector representation h is obtained through BERT. keyword , the keyword text loss L is calculated in the same way keyword

[0139] Step 3-4: Joint loss calculation

[0140] Loss = Ldoc + αL keyword

[0141] The present invention provides an idea and method for an unbalanced text classification method that introduces keyword features. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A method for unbalanced text classification by introducing keyword features, characterized in that: The steps include: Step 1: Build a text classification model, define a hierarchical classification system, build category labels, and build a training data set; Step 2, extract keywords of each category in the text to obtain keyword information; Step 3: Integrate keyword information and text semantic information to perform model training; Wherein, step 2 comprises: Step 2-1: Select category keywords based on normalized point mutual information; Step 2-2: Select category keywords based on improved information gain IG+; Step 2-3: Select a set of keywords for each category; The method for selecting category keywords based on normalized point mutual information in step 2-1 includes: The text in the training data set constructed in step 1-2 is segmented to form a word sequence, and the occurrence of each word and text category is counted. The normalized point mutual information npmi(w,y) between word w and category y is calculated as follows: Where p(w,y) represents the number of co-occurrences of word w and category y divided by the total number of training texts; p(w) and p(y) represent the total number of word and category occurrences divided by the total number of training texts, respectively; The method for selecting category keywords based on improved information gain in step 2-2 includes: The intra-class dispersion degree DI and the inter-class concentration degree CI are introduced, and the improved information gain IG+ method is used to calculate the IG+ score IG+(w,c) of the keywords of the category. The method includes: IG+(w,c)=(DI(w,c)-CI(w,c))IG(w,c) Among them, DI represents the degree of dispersion within the class, which means the ratio of the number of occurrences of word w in the text of category c to the total number of occurrences of the text of this category. The calculation method is: Where df(w,c) represents the number of texts in category c that contain word w, and df(c) represents the number of texts in category c; Among them, CI(w,c) represents the degree of inter-class concentration, that is, the ratio of the number of categories in which word w appears to the total number of categories, where |C| represents the total number of categories, and cf(w) represents the number of categories in which word w appears; The information gain IG(w) of word w is used to measure the degree of uncertainty of text classification caused by the presence or absence of word w. The calculation method includes: Where n represents the total number of text categories, i represents the text traversed in each category; p(w) represents the probability of word w appearing in the text; p(C i |w) represents the posterior probability for the word, that is, the text containing word w belongs to analogy C i probability; Indicates the probability that the text does not contain word w; Indicates that the text does not contain word w and belongs to category C i The posterior probability of The method of selecting a keyword set for each category in step 2-3 includes: Traverse each category, calculate the npmi and IG+ scores of each word, and arrange them in descending order. Select the top 10 words with the highest scores as the word representations of the category, and get 20 keyword representations for each category.

2. The unbalanced text classification method using keyword features according to claim 1, characterized in that: Step 1 includes: Step 1-1, define a hierarchical classification system, characterize the hierarchical relationship between categories, separate labels at different levels by symbols, and perform article-level classification; Step 1-2, building a training data set, includes the following steps: Collecting training corpus; the training corpus includes article titles, article contents and hierarchical classification labels of articles; Preprocessing the training corpus; the preprocessing includes: converting traditional Chinese characters into simplified Chinese characters, and converting full-width numbers and full-width letters into half-width numbers and half-width letters; The article title and body are concatenated with a period to determine whether the length exceeds a preset length threshold; if not, the concatenated result is used as the article; if it exceeds the preset length threshold, the article is truncated and the truncated content is used as the article.

3. The unbalanced text classification method using keyword features according to claim 2 is characterized in that: In the model training process of integrating keyword information and text information described in step 3, for each batch: Step 3-1, calculate the text cross entropy loss function within a single batch; Step 3-2, randomly select N C categories, traverse each category, sample K keywords for each category, and concatenate the keywords to form the keyword text of the category; Step 3-3, calculate the cross entropy loss for the keyword text; Step 3-4, calculate the joint loss.

4. The unbalanced text classification method using keyword features according to claim 3 is characterized in that: Step 3-1 calculates the cross entropy loss function for text classification in a single batch, including: After bert segmentation, special tags [CLS] and [EOS] are added to the beginning and end of the original text, recorded as: x doc =[CLS],t1,t2,…t j ,…,t L ,[EOS] Where L is the length of the text, t L Represents the word unit after bert segmentation; for the sequence x doc , after bert feature extraction, calculate the cross entropy loss L doc , methods include: Among them, C represents the number of categories, y j,c is the true label of the sample, is the output sample x of the text classification model j The probability of belonging to category c, j is the sample number.

5. The unbalanced text classification method using keyword features according to claim 4 is characterized in that: Randomly select N as described in step 3-2 C The calculation method of the number of categories Nc includes: N c =max(batch_size,C) Among them, C represents the total number of categories, batch_size represents the number of samples in this batch; for the keyword text x after keyword selection for the current category c, keyword It is expressed as: x keyword =w1,w2…w K Among them, w K It is the word representing category c obtained after feature selection in steps 2-3.

6. The unbalanced text classification method using keyword features according to claim 5, characterized in that: The method for calculating the cross entropy loss for the keyword text in step 3-3 includes: For keyword text x keyword According to the method described in step 3-1, after BERT segmentation, its semantic vector representation h is obtained. keyword , and calculate the cross entropy loss L after the keyword according to step 3-1 keyword ; Step 3-4 calculates the joint loss based on the text and keywords. The calculation method includes: Loss=L doc +αL keyword Among them, L doc represents the cross entropy loss of the text, L keyword Represents the cross entropy loss of keywords, α is a hyperparameter with a value between 0 and 1.

Images