Text classification method, readable storage medium and electronic device
By constructing and optimizing a text classification model, and utilizing causal reasoning and data augmentation methods, the problems of insufficient generalization ability and robustness of existing models are solved, thereby improving the stability and accuracy of text classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CAS OF CHENGDU INFORMATION TECH CO LTD
- Filing Date
- 2022-11-18
- Publication Date
- 2026-07-31
AI Technical Summary
Existing text classification models have poor generalization ability and robustness, especially when testing out-of-domain data and facing adversarial attacks, their performance deteriorates.
By constructing an initial text classification model, extracting feature words, training a word classifier using a feature word matching algorithm, calculating the similarity of feature words, removing false related words, optimizing the final text classification model, and combining causal inference and data augmentation methods, the robustness of the model is improved.
It achieves more robust text classification, reduces the impact of spurious relevance, and improves the model's performance on different domain data.
Smart Images
Figure CN115935228B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of natural language processing, and more specifically, to a text classification method, a readable storage medium, and an electronic device. Background Technology
[0002] The deployment of large-scale pre-trained models such as BERT (Bidirectional Encoder Representation from Transformers) is a significant milestone in the development of Natural Language Processing (NLP), with large models achieving tremendous progress in real-world applications. As research deepens, the performance of pre-trained models on various NLP tasks continues to improve, but existing models still suffer from limitations such as weak generalization ability and poor robustness. For example, when testing with out-of-domain data, dealing with adversarial attacks, or experiencing minor perturbations at the input, the performance of pre-trained models can drop significantly, impacting the practical deployment of large models. Summary of the Invention
[0003] The purpose of this disclosure is to provide a text classification method, a readable storage medium, and an electronic device to solve the technical problems existing in the prior art, such as the poor generalization ability and robustness of existing text classification models.
[0004] To achieve the above objectives, the first aspect of this disclosure provides a text classification method, including: Based on the text data to be classified, construct an initial text classification model; Extract multiple feature words from the initial text classification model; A word classifier is trained using the aforementioned multiple feature words and feature word matching algorithms; Calculate the feature similarity of each feature word, and determine whether the classification feature of each feature word is a false related word or a true related word; The false related words are removed from the initial text classification model using the word classifier to obtain the final text classification model.
[0005] Optionally, based on the text data to be classified, an initial text classification model is constructed, including: Obtain the text data to be classified, and process the text data to be classified to obtain a training set; An initial text classifier is trained using the training set; wherein the initial text classifier includes: an input layer for inputting the text data; a BERT pre-trained model for encoding the input text data to obtain word vectors, obtaining an input sequence based on the word vectors, and obtaining word importance weights based on the sequence encoded by the BERT pre-trained model.
[0006] Optionally, importance weight analysis is performed on the word vectors encoded by the BERT pre-trained model, including: Given dataset },in, This represents a piece of text data. These are the corresponding tags; Each text data After encoding by the BERT pre-trained model, the CLS flag bit vector of the last layer is taken. The flag bit vector It integrates the semantic information of each word in the text data; calculate With each similarity , , This refers to word vectors at other positions, where similarity is used. Used to represent the importance weight of words.
[0007] Optionally, multiple feature words are extracted from the initial text classification model, including: Based on the importance weight of the words, the top few words with the highest importance weights are identified for each text data and used as feature words to form a feature word set.
[0008] Optionally, the method further includes: The first three words of each text data are extracted as feature words according to the importance weight of the words, thus forming the feature word set.
[0009] Optionally, the feature word matching algorithm includes: For a given text data s, if s contains the word w, denoted as w. If s does not contain the word w, it is denoted as The goal of matching is to find another piece of text. , making The semantics of s and the text data after removing the word w Semantic similarity; calculate the average treatment effect ATE of word w in N text data. N is a positive integer greater than or equal to 2; if word w If the value is greater than or equal to the preset threshold, then word w is a truly relevant word.
[0010] Optionally, the method further includes: The word classifier is optimized by calculating feature parameters, which include one or more of the following: the average of the contextual similarity of multiple closest matches, the standard deviation of the similarity score, and the weighted average treatment effect, wherein the weights in the weighted average treatment effect are calculated from the similarity score.
[0011] Optionally, the method further includes: when testing the final text classification model, dividing the test data into two groups, the first group of test data containing text data with spurious associations that may mislead the text classifier, and the second group of test data containing text data with spurious associations that help the text classifier.
[0012] A second aspect of this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect.
[0013] A third aspect of this disclosure provides an electronic device, comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method described in the first aspect.
[0014] This disclosure addresses the shortcomings of text classification models that typically focus only on statistical correlation while neglecting causal correlation. It provides a text classification method based on the BERT model and data augmentation, considering the potential causal relationships in the text. The data augmentation method draws on counterfactual theory in the field of causal inference, mainly solving the problem of weak generalization ability and robustness of existing text classification models. Compared with the current prior art, the classification model of the scheme in this disclosure will achieve more robust classification and reduce the impact of spurious correlations in text classification.
[0015] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description
[0016] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:
[0017] Figure 1 This is a flowchart illustrating a text classification method according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating a word classifier construction and data adjustment scheme according to an exemplary embodiment; Figure 3This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation
[0018] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0019] In this embodiment, causal inference is used to eliminate confounding factors with spurious correlations, effectively improving model robustness. Causal inference is an important research topic in many fields such as statistics, computer science, and economics, and causal methods can be used to solve robustness problems. Existing causal methods use artificial causal knowledge for data augmentation to make the model more robust under out-of-domain data distributions. However, real-world data often contains confounding factors, namely, the causes and variables of potential influences. These unmeasured or potential confounding factors may bias the model. Therefore, using causal inference to eliminate confounding factors with spurious correlations can effectively improve model robustness.
[0020] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a text classification method according to an exemplary embodiment, which is used to improve the robustness of text classification and includes the following steps.
[0021] Step 101: Construct an initial text classification model based on the text data to be classified; Step 102: Extract multiple feature words from the initial text classification model; Step 103: Train a word classifier using the multiple feature words and the feature word matching algorithm; Step 104: Calculate the feature similarity of each feature word and determine whether the classification feature of each feature word is a false related word or a true related word; Step 105: Use the word classifier to remove the false related words from the initial text classification model to obtain the final text classification model.
[0022] While many current text classification models have achieved state-of-the-art performance on benchmarks, they still struggle to ensure robustness in unseen or challenging scenarios. This disclosure addresses the shortcomings of text classification models, which typically focus only on statistical correlations while neglecting causal correlations. It provides a text classification method based on the BERT model and data augmentation, considering potential causal relationships within the text. The data augmentation method draws upon counterfactual theory from the field of causal inference, primarily addressing the weak generalization ability and robustness of existing text classification models. Compared to current techniques, this method achieves more robust classification and reduces the impact of spurious correlations in text classification.
[0023] Next, combined Figure 1 and Figure 2 The solutions in the embodiments of this disclosure will be described.
[0024] In this embodiment of the disclosure, based on the text data to be classified, the text data is processed to obtain a suitable training set. An initial text classifier is trained using the training set. The initial text classifier includes: an input layer for inputting the text data; and a BERT pre-trained model for encoding the input text data to obtain word vectors and sentence vectors. Then, an input sequence is obtained based on the word vectors, and the importance weights of the words are obtained according to the sequence encoded by the BERT pre-trained model.
[0025] The importance weight analysis process for word vectors encoded by the BERT model is as follows: Given a dataset },in This represents a piece of text data. These are the corresponding tags; each piece of text data After BERT encoding, the CLS flag bit vector of the last layer is taken. This vector integrates the semantic information of each word in the text; assuming ,calculate With each similarity, Similarity It also indicates the importance weight of words.
[0026] Then, based on the vector, the importance weights of the words are obtained, and for each training sample, the top-weighted words are identified as feature words; in this embodiment, the similarity can be used as a basis for further analysis. The first three words of each training sample are extracted as feature words and a feature word set is formed.
[0027] In one exemplary embodiment, drawing on the counterfactual theory of causal reasoning, a feature word matching algorithm is designed using a feature word set to train a word classifier. Specifically, this includes: for a given text segment s, if it contains the word w, denoted as... If there is no word 'w', it is denoted as The goal of matching is to find another piece of text. This makes its semantics consistent with the original text data after removing the word 'w', thus making it consistent with the original text data. The semantics should be as similar as possible. Through this optimal matching process, the average treatment effect (ATE) of word w in N text data can be calculated: If word w is a real related word, then w will have a greater [positive value]. The value is used as the primary basis for classification. In this embodiment of the disclosure, a relatively large value can be set. The value is used as a preset threshold; if w If the value is greater than or equal to the preset threshold, then w is determined to be a real related word.
[0028] In another exemplary embodiment, for the word classifier, to address the problem that high-quality matches may not exist in the data, analogous to supervised learning methods for causal inference, instead of directly using ATE to distinguish between spurious and true relevance, more feature parameters are calculated to optimize the word classifier. For example, the feature parameters may be one or more of the following: the average of the contextual similarity of multiple closest matches, the standard deviation of the similarity score, and the weighted average treatment effect; wherein, the weights in the weighted average treatment effect are calculated from the similarity score, and the average of the contextual similarity of multiple closest matches may be the average of the contextual similarity of the top 5 closest matches or the average of the contextual similarity of the top 3 closest matches.
[0029] Then, a word classifier is applied to optimize text classification, spurious related words obtained by the word classifier are removed from the original text, and the final text classification model is trained.
[0030] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating a word classifier construction and data adjustment scheme according to an exemplary embodiment. In this embodiment, since the same spurious relevance may exist in both training and testing data, removing spurious words does not necessarily improve the accuracy of the classification model on random samples. To demonstrate the effectiveness of this embodiment, it is necessary to test the robustness improvement effect of the model. Specifically, the test data is divided into two groups, and the model performance of each group is explored. The first group of test data contains text that spurious associations may mislead the classifier, and the second group of test data contains text that spurious associations help the classifier.
[0031] Figure 3 This is a block diagram illustrating an electronic device 300 according to an exemplary embodiment. Figure 3 As shown, the electronic device 300 may include a processor 301 and a memory 302. The electronic device 300 may also include one or more of a multimedia component 303, an input / output (I / O) interface 304, and a communication component 305.
[0032] The processor 301 controls the overall operation of the electronic device 300 to complete all or part of the steps described above. The memory 302 stores various types of data to support the operation of the electronic device 300; this data may include, for example, instructions for any application or method operating on the electronic device 300, and application-related data. The multimedia component 303 may include a screen and audio components. The I / O interface 304 provides an interface between the processor 301 and other interface modules. The communication component 305 is used for wired or wireless communication between the electronic device 300 and other devices.
[0033] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the text classification method described above. For example, the computer-readable storage medium may be the memory 302 including the program instructions described above, which may be executed by the processor 301 of the electronic device 300 to complete the text classification method described above.
[0034] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described text classification method when executed by the programmable device.
[0035] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.
[0036] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.
[0037] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.
Claims
1. A method of text classification, characterized by, include: Based on the text data to be classified, an initial text classification model is constructed, including: Obtain the text data to be classified, and process the text data to be classified to obtain a training set; An initial text classifier is trained using the training set; wherein the initial text classifier includes: an input layer for inputting the text data; a BERT pre-trained model for encoding the input text data to obtain word vectors, obtaining an input sequence based on the word vectors, and obtaining word importance weights based on the sequence encoded by the BERT pre-trained model; Multiple feature words are extracted from the initial text classification model, including: Based on the importance weight of the words, for each text data, find the words with the highest importance weight as feature words and form a feature word set; A word classifier is trained using the aforementioned multiple feature words and feature word matching algorithms; Calculate the feature similarity of each feature word, and determine whether the classification feature of each feature word is a false related word or a true related word; The false related words are removed from the initial text classification model using the word classifier to obtain the final text classification model; The importance weight analysis of the word vectors encoded by the BERT pre-trained model includes: Given dataset wherein, denotes a piece of text data, is the corresponding label; Each piece of text data After encoding by the BERT pre-training model, the last layer CLS flag bit vector is taken The flag bit vector Fuses the semantic information of each word in the text data Computing Similarity to each , , is a vector of other positions, similarity is used to represent the importance weight of the word The feature word matching algorithm includes: For a given text data s, if s contains the word w, denoted as w. If s does not contain the word w, it is denoted as The goal of matching is to find another piece of text. , making The semantics of s and the text data after removing the word w Semantic similarity; calculate the average treatment effect ATE of word w in N text data. N is a positive integer greater than or equal to 2; if word w If the value is greater than or equal to the preset threshold, then word w is a truly relevant word.
2. The text classification method as described in claim 1, characterized in that, The method further includes: The first three words of each text data are extracted as feature words according to the importance weight of the words, thus forming the feature word set.
3. The text classification method as described in claim 1, characterized in that, The method further includes: The word classifier is optimized by calculating feature parameters, which include one or more of the following: the average of the contextual similarity of multiple closest matches, the standard deviation of the similarity score, and the weighted average treatment effect, wherein the weights in the weighted average treatment effect are calculated from the similarity score.
4. The text classification method of claim 1, wherein, The method further includes: when testing the final text classification model, dividing the test data into two groups, the first group of test data containing text data with spurious associations that may mislead the text classifier, and the second group of test data containing text data with spurious associations that help the text classifier.
5. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the steps of the text classification method according to any one of claims 1-4.
6. An electronic device, comprising: include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the text classification method according to any one of claims 1-4.