Weakly supervised text classification system and method based on class name guidance

Abstract

The present invention relates to a weakly supervised text classification system and method based on class name guidance. A seed word generation module learns vector representation, modeling relationship and generates seed words for corpus in a corpus; a pseudo-label assignment module assigns predicted pseudo-labels to documents in the corpus; a document classifier module performs pre-training based on documents assigned pseudo-labels and generalizes training on unlabeled documents; a seed word extension module extends seed words by comprehensively considering the classification results and the ranking scores generated when the seed words are generated after a classification. First, the class name and the corpus are used to generate seed words; the seed words after the seed words are generated are iteratively applied to pseudo-label assignment, and the document classifier is trained using documents assigned pseudo-labels, and the seed words are extended by the predicted results; the classification results are output after the number of iterations is completed. The reliance on high-quality annotated corpora is reduced to solve the problem of data scarcity in the field of text classification.

Description

Technical Field

[0001] The invention relates to a weakly supervised text classification system and method based on class name guidance. Background Art

[0002] At present, text classification is the most basic task in natural language processing, and it can be seen in everything from sentiment analysis to intent recognition and information extraction. In recent years, recurrent neural networks, convolutional neural networks, and BERT pre-trained models have achieved excellent results in text classification tasks. These text classification models are becoming more and more popular in academia. To train a supervised classification model, at least hundreds of thousands of annotated documents are required. However, collecting training documents of this scale often requires the cooperation of a large number of annotators and document experts. The lack of training data is an important reason why supervised classification models are difficult to implement on a large scale.

[0003] The main technical methods of weakly supervised text classification are divided into: class name-based methods, seed word-based methods, etc. Class name-based methods mostly use the method of embedding labels and documents into semantic space, calculating the semantic similarity between documents and potential labels as the main means, and using logistic regression models for classification; seed word-based methods mostly use seed words provided by experts who are particularly familiar with the corpus as a supervision source, and use contextualization technology for text classification, such as: BERT model, ELMo model, etc. Existing weakly supervised text classification technologies have many limitations. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a weakly supervised text classification system and method based on class name guidance.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] The weakly supervised text classification system based on class name guidance is characterized by: including a seed word generation module, a pseudo label assignment module, a document classifier module and a seed word expansion module;

[0007] The seed word generation module learns vector representations, models relationships, and generates seed words for the corpus;

[0008] The pseudo-label assignment module assigns predicted pseudo-labels to documents in the corpus;

[0009] The document classifier module is pre-trained on documents that have been assigned pseudo labels and generalized on unlabeled documents;

[0010] After a classification is completed, the seed word expansion module expands the seed words by comprehensively considering the classification results and the ranking scores generated when the seed words are generated.

[0011] Furthermore, in the above-mentioned weakly supervised text classification system guided by class name, the seed word generation module includes a generation module and a selection module. The generation module is used to learn vector representation and modeling relationship, model the class name and the documents in the corpus, and obtain a series of semantically related words through similarity calculation; the selection module is used to generate seed words, and the obtained semantically related words are combined with semantic specificity to obtain word ranking scores, so as to generate high-quality seed words.

[0012] Furthermore, in the above-mentioned weakly supervised text classification system guided by class name, the pseudo-label assignment module counts the frequency of occurrence of high-quality seed words generated by the seed word generation module in the document, and assigns pseudo-labels to unlabeled documents in combination with the word ranking scores obtained by the seed word generation module.

[0013] Furthermore, in the above-mentioned weakly supervised text classification system guided by class name, the document classifier module adopts a hierarchical attention model as a classifier, first focuses on the sentences in the document, finds the important sentences in the document, and then focuses on the words in the sentence to identify the important words in the sentence; then the pseudo-labels generated by the pseudo-label assignment module are used to train a HAN model on the unlabeled document data.

[0014] Furthermore, in the above-mentioned weakly supervised text classification system guided by class name, the seed word expansion module calculates the expansion score by comprehensively considering the predicted probability of the document obtained by the document classifier module and the frequency of occurrence of the seed word in the predicted document; the seed word expansion module expands the seed word set of the seed word generation module, assigns pseudo labels again through the pseudo label assignment module, and then trains the document classifier module until convergence.

[0015] The present invention is based on a weakly supervised text classification method guided by class name, which includes the following steps:

[0016] First, the class name and the corpus are used to generate seed words;

[0017] Then, the seed words after seed word generation are iteratively applied to pseudo-label assignment, and the document classifier is trained using the documents assigned with pseudo-labels, and the seed words are expanded through the predicted results;

[0018] Finally, wait until the number of iterations is over and output the classification results.

[0019] Furthermore, in the above-mentioned weakly supervised text classification method guided by class name, a seed word generation module learns vector representation, modeling relationship, and generates seed words for the corpus; a pseudo-label assignment module assigns predicted pseudo-labels to documents in the corpus; a document classifier module performs pre-training based on documents with assigned pseudo-labels, and then generalizes the training on unlabeled documents; and a seed word expansion module expands the seed words after a classification is completed by comprehensively considering the classification results and the ranking scores generated when generating the seed words.

[0020] Furthermore, in the above-mentioned weakly supervised text classification method guided by class name, the generation module of the seed word generation module models the class name with the document in the corpus, and obtains a series of semantically related words through similarity calculation; the selection module of the seed word generation module jointly considers the semantic specificity of the obtained semantically related words to obtain a word ranking score, so as to generate high-quality seed words;

[0021] The generation module in the seed word generation module uses von Misser distribution to model the class name and corpus, and uses similarity calculation to obtain a set of words, which is calculated as follows:

[0022] von Misser distribution, x w is the m-dimensional vector of the words in the corpus, is the modified Bessel function of the first kind of order; the von Misser distribution has two parameters, the mean direction and the concentration parameter, and the label name vector u l is taken as the average direction, and the concentration of other words near the tag name is k l Being taken as a concentration parameter, the distribution expression is:

[0023]

[0024] Similarity calculation, cosine similarity can extract the semantic correlation between words, thereby obtaining a series of words related to the class name, and obtain a set of words W that are highly semantically related to the label name l through vector cosine similarity. k :

[0025]

[0026]

[0027] Von Misser distribution is a spherical distribution. Words that are semantically related to the central word are gathered around the central word. By calculating the similarity, a series of words related to the class name can be obtained.

[0028] The selection module of the seed word generation module uses semantic specificity and combines similarity calculation to obtain the final seed word. The calculation is as follows:

[0029] Semantic specificity: if the meaning of word v contains the meaning of another word w, then all the contextual features of word v will appear in word w; use scalar SC w，l Associate word w with label name l, SC w，l When the value is larger, it indicates that the semantics of word w is more specific and exclusive than that of tag name l. w，l The values ​​are normalized to obtain the semantic specificity score:

[0030]

[0031] The pseudo-label assignment module assigns predicted pseudo-labels to documents in the corpus;

[0032] The document classifier module is pre-trained on documents that have been assigned pseudo labels, and then generalized on unlabeled documents;

[0033] After a classification is completed, the seed word expansion module expands the seed words by comprehensively considering the classification results and the ranking scores generated when the seed words are generated;

[0034] The seed word expansion module expands the seed word set of the seed word generation module, assigns pseudo labels again through the pseudo label assignment module, and then trains the document classifier module until convergence; the iterative training framework gives it stronger generalization ability and effectiveness.

[0035] Compared with the prior art, the present invention has significant advantages and beneficial effects, which are specifically embodied in the following aspects:

[0036] ① The weakly supervised text classification system and method guided by class names in the present invention learns vector representations, models relationships, and generates seed words for the corpus. Through the seed word generation module, seed words related to the class name are generated, which effectively reduces the dependence on high-quality annotated corpora and solves the problem of data scarcity in the field of text classification.

[0037] ② The method of combining class name and seed word is used to replace the traditional method based solely on class name and seed word. This invention innovatively solves the problem of data scarcity in text classification and has higher classification accuracy on general data sets than the existing technology.

[0038] ③ Carry out practical research on text classification methods with extremely high application value, which is pioneering and significantly improves the accuracy of successful classification.

[0039] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by practicing the specific embodiments of the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 : Schematic diagram of the architecture principle of the system of the present invention;

[0042] Figure 2 : Schematic diagram of the process of the present invention;

[0043] Figure 3 : Schematic diagram of the architecture principle of the seed word generation module;

[0044] Figure 4 : Schematic diagram of the architecture principle of the pseudo-label assignment module;

[0045] Figure 5 : Schematic diagram of the architecture of the document classifier module;

[0046] Figure 6 : Schematic diagram of the architecture principle of the seed word expansion module;

[0047] Figure 7 :Generate effect display diagram. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0049] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, directional terms and order terms are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0050] like Figure 1 As shown, the weakly supervised text classification system based on class name guidance includes a seed word generation module 1, a pseudo label assignment module 2, a document classifier module 3 and a seed word expansion module 4;

[0051] Seed word generation module 1, which learns vector representation, models relationships, and generates seed words for the corpus;

[0052] Pseudo-label assignment module 2, assigns predicted pseudo-labels to documents in the corpus;

[0053] Document classifier module 3, pre-trained on documents with assigned pseudo labels and generalized on unlabeled documents;

[0054] The seed word expansion module 4 expands the seed words by comprehensively considering the classification results and the ranking scores generated when the seed words are generated after a classification is completed.

[0055] like Figure 3 The seed word generation module 1 includes a generation module 101 and a selection module 102. The generation module 101 is used to learn vector representation and modeling relationship, model the class name and the document in the corpus, and obtain a series of semantically related words through similarity calculation; the selection module 102 is used to generate seed words, and the obtained semantically related words are combined with semantic specificity to obtain word ranking scores, so as to generate high-quality seed words.

[0056] like Figure 4 ,The pseudo-label assignment module 2 counts the frequency of the high-quality seed words generated by the seed word generation module 1 in the document, and combines the word ranking scores obtained by the seed word generation module 1 to assign pseudo-labels to the unlabeled documents.

[0057] like Figure 5 ,Document classifier module 3,adopts the hierarchical attention model as the classifier,firstly focuses on the sentences in the document to find the important sentences in the document;then focuses on the words in the sentence to identify the important words in the sentence.,Then use the pseudo labels generated by the pseudo label assignment module 2 to train a HAN model on the unlabeled document data.

[0058] like Figure 6The seed word expansion module 4 calculates an expansion score by comprehensively considering the predicted probability of the document obtained by the document classifier module 3 and the frequency of occurrence of the seed word in the predicted document, so as to better expand the seed word.

[0059] The present invention provides a weakly supervised text classification method and system based on class name guidance, comprising the following steps:

[0060] First, the class name and the corpus are used to generate seed words;

[0061] Then, the seed words after seed word generation are iteratively applied to pseudo-label assignment, and the document classifier is trained using the documents assigned with pseudo-labels, and the seed words are expanded through the predicted results;

[0062] Finally, wait until the number of iterations is over and output the classification results.

[0063] like Figure 2 , the seed word expansion module 1 learns vector representation, models relationship and generates seed words for the corpus; the pseudo-label assignment module 2 assigns predicted pseudo-labels to the documents in the corpus; the document classifier module 3 performs pre-training based on the documents with assigned pseudo-labels, and then generalizes the training on unlabeled documents; the seed word expansion module 4, after a classification is completed, expands the seed words by comprehensively considering the classification results and the ranking scores generated when generating the seed words.

[0064] The generation module 101 of the seed word generation module 1 models the class name with the documents in the corpus, and obtains a series of semantically related words through similarity calculation; the selection module 102 of the seed word generation module 1 combines the obtained semantically related words with semantic specificity to obtain a word ranking score, so as to generate high-quality seed words;

[0065] The generation module 101 in the seed word generation module 1 uses the von Misser distribution to model the class name and the corpus, and uses similarity calculation to obtain a set of words, which is calculated as follows:

[0066] von Misser distribution, x w is the m-dimensional vector of the words in the corpus, is the modified Bessel function of the first kind of order; the von Misser distribution has two parameters, the mean direction and the concentration parameter, and the label name vector u l is taken as the average direction, and the concentration of other words near the tag name is k l Being taken as a concentration parameter, the distribution expression is:

[0067]

[0068] Similarity calculation, cosine similarity can extract the semantic correlation between words, thereby obtaining a series of words related to the class name, and obtain a set of words W that are highly semantically related to the label name l through vector cosine similarity. k :

[0069]

[0070]

[0071] Von Misser distribution is a spherical distribution. Words that are semantically related to the central word are gathered around the central word. By calculating the similarity, a series of words related to the class name can be obtained.

[0072] The selection module 102 of the seed word generation module 1 uses semantic specificity and combines similarity calculation to obtain the final seed word, which is calculated as follows:

[0073] Semantic specificity: if the meaning of word v contains the meaning of another word w, then all the contextual features of word v will appear in word w; use scalar SC w，l Associate word w with label name l, SC w，l When the value is larger, it indicates that the semantics of word w is more specific and exclusive than that of tag name l. w，l The values ​​are normalized to obtain the semantic specificity score:

[0074]

[0075] The pseudo-label assignment module 2 assigns predicted pseudo-labels to the documents in the corpus;

[0076] The document classifier module 3 is pre-trained on documents that have been assigned pseudo labels, and then generalized on unlabeled documents;

[0077] After a classification is completed, the seed word expansion module 4 expands the seed word by comprehensively considering the classification result and the ranking score generated when the seed word is generated;

[0078] The seed word expansion module 4 expands the seed word set of the seed word generation module, assigns pseudo labels again through the pseudo label assignment module 2, and then trains the document classifier module 3 until convergence; the iterative training framework gives it stronger generalization ability and effectiveness.

[0079] like Figure 7As shown, for example, the example: "Under the terms of the deal, shareholders will receive $54.20 in cash for each share of Twitter stock they own, matching Musk's original offer and marking a 38% premium over the stock price the day before Musk revealed his stake in the company." is not labeled, but should belong to the Business category. By using the weakly supervised text classification system guided by the class name of the present invention, effective information can be extracted from the text data step by step, and the correct result can be finally output.

[0080] The user provides the class name, the class name and the corpus are modeled, and the seed word generation operation is performed; then, the generated seed words are applied to pseudo-label assignment, the document classifier is trained using the documents assigned with pseudo-labels, and the seed words are expanded through the predicted results; the expanded seed words are applied again to pseudo-label assignment, and then the document classifier is trained; finally, the classification results are output after the method converges.

[0081] In summary, the weakly supervised text classification system and method guided by class names learns vector representations, models relationships, and generates seed words for the corpus. Through the seed word generation module, seed words related to the class name are generated, which effectively reduces the dependence on high-quality annotated corpora and solves the problem of data scarcity in the field of text classification. The method of combining class names and seed words replaces the traditional methods based solely on class names and seed words; the present invention innovatively solves the problem of data scarcity in text classification, and has a higher classification accuracy rate on general data sets than the prior art; it is pioneering to carry out practical research on text classification methods with extremely high application value, and significantly improves the accuracy rate of successful classification.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention. It should be noted that similar numbers and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0083] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0084] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A weakly supervised text classification method based on class name guidance, characterized by: The following steps are involved: First, the class name and the corpus are used to generate seed words; Then, the seed words after seed word generation are iteratively applied to pseudo-label assignment, and the document classifier is trained using the documents assigned with pseudo-labels, and the seed words are expanded through the predicted results; Finally, wait until the number of iterations ends and output the classification results; The generation module (101) of the seed word generation module (1) models the class name with the document in the corpus, and obtains a series of semantically related words by similarity calculation; the selection module (102) of the seed word generation module (1) combines the obtained semantically related words with semantic specificity to obtain a word ranking score, so as to generate high-quality seed words; The generation module (101) in the seed word generation module (1) uses the von Misser distribution to model the class name and the corpus, and uses similarity calculation to obtain a set of words, which is calculated as follows: von Misser distribution, x w is the m-dimensional vector of the words in the corpus, yes Modified Bessel function of the first kind of order; The von Misser distribution has two parameters: the mean direction and the concentration parameter. The label name vector u l is taken as the average direction, and the concentration of other words near the tag name is k l Being taken as a concentration parameter, the distribution expression is: Similarity calculation, cosine similarity can extract the semantic correlation between words, thereby obtaining a series of words related to the class name, and obtain a set of words W that are highly semantically related to the label name l through vector cosine similarity. k : Von Misser distribution is a spherical distribution. Words that are semantically related to the central word are gathered around the central word. By calculating the similarity, a series of words related to the class name can be obtained. The selection module (102) of the seed word generation module (1) uses semantic specificity and combines similarity calculation to obtain the final seed word, which is calculated as follows: Semantic specificity: if the meaning of word v contains the meaning of another word w, then all the contextual features of word v will appear in word w; use scalar SC w，l Associate word w with label name l, SC w，l When the value is larger, it indicates that the semantics of word w is more specific and exclusive than that of tag name l. w，l The values ​​are normalized to obtain the semantic specificity score: The pseudo-label assignment module (2) assigns predicted pseudo-labels to the documents in the corpus; The document classifier module (3) is pre-trained on documents that have been assigned pseudo-labels, and then generalized on unlabeled documents; The seed word expansion module (4) expands the seed words by comprehensively considering the classification results and the ranking scores generated when the seed words are generated after a classification is completed; The seed word expansion module (4) expands the seed word set of the seed word generation module (1), assigns pseudo labels again through the pseudo label assignment module (2), and then trains the document classifier module (3) until convergence.

2. The weakly supervised text classification method based on class name guidance according to claim 1 is characterized in that: The seed word generation module (1) comprises a generation module (101) and a selection module (102), wherein the generation module (101) is used to learn vector representation and model relationship, model the class name and the documents in the corpus, and obtain a series of semantically related words through similarity calculation; The selection module (102) is used to generate seed words, combine the obtained semantically related words with semantic specificity, and obtain word ranking scores, so as to generate high-quality seed words.

3. The weakly supervised text classification method based on class name guidance according to claim 1 is characterized in that: The pseudo-label assignment module (2) counts the frequency of occurrence of high-quality seed words generated by the seed word generation module (1) in the document, and assigns pseudo-labels to unlabeled documents in combination with the word ranking scores obtained by the seed word generation module (1).

4. The weakly supervised text classification method based on class name guidance according to claim 1 is characterized in that: The document classifier module (3) uses a hierarchical attention model as a classifier, first focusing on the sentences in the document to find the important sentences in the document, and then focusing on the words in the sentences to identify the important words in the sentences; then the pseudo-labels generated by the pseudo-label assignment module (2) are used to train a HAN model on the unlabeled document data.

5. The weakly supervised text classification method based on class name guidance according to claim 1 is characterized in that: The seed word expansion module (4) calculates an expansion score by comprehensively considering the predicted probability of the document obtained by the document classifier module (3) and the frequency of occurrence of the seed word in the predicted document; the seed word expansion module (4) expands the seed word set of the seed word generation module (1), assigns pseudo labels again through the pseudo label assignment module (2), and then trains the document classifier module (3) until convergence.

Images