Text classification method based on ensemble learning

Abstract

The invention relates to a text classification method based on ensemble learning, which belongs to the field of text classification, and comprises the following steps: collecting samples, marking categories, and constructing a data set; preprocessing the data in the text; selecting three base classifiers and adopting a tandem training method, wherein each base classifier depends on a feature vector generated by the previous base classifier in the training process; training the training data set, wherein each base classifier outputs a corresponding confidence coefficient; searching a weight adaptive to each base classifier on the verification data set, calculating and adjusting the weight by adopting a particle swarm optimization (PSO) algorithm, and performing weighted fusion on the confidence coefficient by utilizing the optimal weight; and obtaining a classification result of the text data according to the fused confidence coefficients, and taking the category with the highest confidence coefficient in the obtained multiple fused confidence coefficients as a prediction classification result. The method provided by the invention can realize multi-dimensional extraction and effective utilization of features, and has higher classification accuracy and model generalization ability.

Description

Technical Field

[0001] The invention belongs to the field of text classification, and in particular relates to a text classification method based on ensemble learning. Background Art

[0002] With the continuous development of Internet big data, automatic and accurate classification of text data has become an urgent need for data processing and data analysis. Text classification methods based on deep learning technology have shown better classification results than previous rule-based or traditional machine learning methods. Currently popular deep learning network structures such as TextCNN, TextRNN, FastText, Transformer, etc. have their own strengths in text classification tasks, and have advantages in extracting text features in terms of text data length, text information association distance, etc. However, the generalization ability of a single neural network is weak, and the accuracy of text classification needs to be improved.

[0003] Ensemble learning technology can effectively combine the complementary information extracted from different deep learning neural networks. By selecting several base classifiers with good performance and complementary advantages and choosing appropriate combination strategies, it can achieve higher prediction accuracy than a single base classifier and improve the reliability and robustness of the text classification model.

[0004] At present, common ensemble learning methods include voting, averaging, bagging, boosting and stacking. Existing ensemble learning methods have the following problems:

[0005] 1. The classification accuracy of a single text classifier is low and the model generalization ability is insufficient;

[0006] 2. The existing text classification models do not extract and utilize text feature information to a high degree;

[0007] 3. The model fusion method used in existing ensemble learning lacks dynamic weighted adjustment and the weight calculation is unreasonable.

[0008] In summary, existing ensemble learning methods lack dynamic adjustment functions in the combination strategy of base classifiers. The ability to assign different weights to different base classifiers based on their advantages and results is an important direction to further improve the dynamism and robustness of ensemble methods. Therefore, how to improve the classification accuracy and generalization ability of text classification models is a technical problem that needs to be solved at present. Summary of the invention

[0009] In view of the above deficiencies in the prior art, the purpose of the present invention is to provide a text classification method based on ensemble learning, which realizes multi-dimensional extraction and effective utilization of features and has higher classification accuracy and model generalization ability.

[0010] The present invention proposes a text classification method based on ensemble learning, comprising:

[0011] S1, collect samples and mark categories, build a data set, and divide the data set into a training data set, a validation data set, and a test data set in a preset ratio;

[0012] S2, preprocessing the data of this article, including text standardization, text cleaning, word segmentation, word cleaning and normalization, dictionary building and word embedding;

[0013] S3, selecting three base classifiers, adopting a serial training method for the three base classifiers, each base classifier depends on the feature vector generated by the previous base classifier during the training process; inputting a training data set, training the training data set in sequence according to the three serially connected base classifiers, and each of the base classifiers outputs a corresponding confidence level;

[0014] S4, using the confidences output by the three base classifiers as input data, searching for weights suitable for each base classifier on the validation data set, and using a particle swarm optimization (PSO) algorithm to calculate and adjust the weights, and determining the optimal weight of each base classifier through iterative feedback, and using the optimal weights to perform weighted fusion on the corresponding confidences to obtain fused confidences;

[0015] S5, obtaining the classification result of the text data according to the fused confidence, and taking the category with the highest confidence among the multiple fused confidences obtained as the predicted classification result.

[0016] Further, in S1, the data set is divided into a training data set, a validation data set and a test data set in a ratio of 6:2:2.

[0017] Furthermore, in S2, the data in this article is cleaned and normalized, including: removing stop words, part-of-speech tagging, word form restoration, spelling correction, extracting valid words, and restoring the part of speech and tense of the words;

[0018] The dictionary is established for the data in this article, including: firstly, counting and encoding the words appearing in the corpus to generate a dictionary, and then converting the words into corresponding codes to preliminarily realize the digital representation of the text.

[0019] The data in this article is embedded in words, including: segmenting the data after text preprocessing at the character and word granularity, and constructing character embedding and word embedding.

[0020] Furthermore, in S3, the three base classifiers are respectively a TextRCNN model, a FastText model and a Transformer model.

[0021] Furthermore, a serial training method is adopted to add the penultimate feature vector of the fully connected layer of the TextRCNN model to the input of the fully connected layer of the FastText model, and the penultimate feature vector of the fully connected layer of the FastText model is added to the input of the linear layer of the Transformer model.

[0022] Furthermore, the TextRCNN model is used to extract local and global features of text and fine-grained text information;

[0023] The FastText model is used to add n-gram vectors to extract continuous word order information, and uses a fully connected neural network structure to capture shallow information of the text;

[0024] The Transformer model is used to add position vectors of words to capture positional relationships.

[0025] Furthermore, in S4, the confidence after fusion is calculated, including:

[0026] (1) Assume there are three base classifiers, and initialize the weight vector W = {w1, w2, w3} with a length of 3;

[0027] (2) The PSO algorithm is used to adjust and optimize the weight vector through iterative calculation to find the optimal weight;

[0028] (3) In the iterative process, the fitness function is used as a criterion for measuring the quality of a particle's position. The PSO algorithm optimizes the particle position by continuously updating and feedback operations, and finally finds the global optimal value gbest as the weight vector for model fusion, determines the optimal weight of each base classifier, and performs weighted fusion based on the optimal weight of each base classifier and the output confidence to obtain the fused confidence.

[0029] Furthermore, the adaptation function adopts binary cross entropy loss,

[0030]

[0031] Among them, fitness is the adaptation function, l ij represents the true value of the i-th sample label j, s ij represents the probability of label j on sample i, N is the number of samples, and Z is the number of labels.

[0032] Furthermore, the confidence after fusion is calculated using the following formula:

[0033]

[0034] Among them, Score is the confidence of the entire integrated model after model fusion, k is the number of base classifiers, k = 3, w i Represents the weight of each base classifier obtained by the PSO algorithm, Score i =[s i1 ,…,s iZ ] represents the confidence of the output of the i-th base classifier, and Z is the number of labels.

[0035] Furthermore, after S5, the method further includes: using the test data set to test the accuracy of the classification result.

[0036] The beneficial effects of the present invention are as follows:

[0037] (1) Text classification method based on ensemble learning: The present invention can extract sufficient vector features from different granularities for text data of different lengths, take advantage of the strengths of the base classification with excellent performance, realize multi-dimensional extraction and effective utilization of features, and further achieve higher classification accuracy and model generalization ability compared with a single classification model.

[0038] (2) Serial training mode of base classifiers: The serial training of the present invention can associate the three base classifiers during the training process, using their respective advantages to increase the richness of features and extract more levels of text feature information.

[0039] (3) Model fusion method based on PSO algorithm: The present invention uses the update feedback mechanism of the PSO algorithm to determine the weights of the three base classifiers, enhances the feedback mechanism of weight calculation, effectively makes up for the lack of dynamic adjustment function in previous fusion strategies, and improves the performance of the classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0041] Figure 1 A flowchart of a text classification method based on ensemble learning according to an embodiment of the present invention;

[0042] Figure 2 This is a diagram of a text classification model based on ensemble learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0044] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0045] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0047] The technical terms involved in the present invention are explained below:

[0048] Text classification: Text classification is a classic task in the field of natural language processing, which annotates a piece of text with a predefined category label.

[0049] Machine Learning (ML): Machine learning is a technology that uses limited observational data to learn and infer general rules and improve its own performance. It relies on pattern recognition and reasoning and uses statistical models and algorithms to achieve specific tasks.

[0050] Deep Learning (DL): A branch of machine learning, it is a feature learning method that converts raw data into a higher-level, more abstract expression through a nonlinear complex model. It uses an artificial neural network as the architecture to perform feature learning on data.

[0051] Ensemble Learning (EL): Ensemble learning is a machine learning technique that combines multiple learners to form a learning model. The text classification method based on ensemble learning completes the classification task by combining several independent base classifiers through a combination strategy, so that the classification model can achieve higher classification accuracy and generalization ability.

[0052] The present invention proposes a text classification method based on ensemble learning. The method selects three base classifiers with good performance and uses ensemble learning technology to complement each other. The method adopts the method of serial training to strengthen the association and feature utilization between the base classifiers. Finally, a model fusion strategy based on the PSO algorithm is proposed to calculate and adjust the weighted weights, thereby improving the robustness and classification accuracy of the text classification model. The method mainly includes the following four stages: data set construction, text preprocessing, model training and model fusion.

[0053] like Figure 1 As shown, the present invention proposes a text classification method based on ensemble learning, comprising the following steps:

[0054] S1, collect samples and mark categories, build a data set, and divide the data set into a training data set, a validation data set, and a test data set in a preset ratio. Among them, the training data set is used for model training, the validation data set is used for model fusion, and the test data set is used to test the classification accuracy of the classification model.

[0055] In the embodiment of the present invention, the data set is divided into a training data set, a validation data set and a test data set in a ratio of 6:2:2. It should be noted that the above ratio can be adjusted and set as needed.

[0056] S2, preprocessing the data of this article, including text standardization, text cleaning, word segmentation, word cleaning and normalization, dictionary building and word embedding.

[0057] Each step is described below.

[0058] Text standardization: Convert all characters to half-width and all English letters to lowercase.

[0059] Text cleaning: remove invalid characters from text.

[0060] Text identifier replacement: A method for identifying and replacing characteristic words in the IT operation and maintenance field is proposed. The character strings that need to be replaced are identified according to the generation rules of specific types of symbols, and replaced with corresponding identifiers.

[0061] Word segmentation: Use NLTK natural language processing tools for word segmentation.

[0062] Word cleaning and normalization: including the steps of removing stop words, part-of-speech tagging, word form restoration, spelling correction, etc., to initially extract valid words and restore the word parts of speech and tenses.

[0063] Build a dictionary: First, count and encode the words that appear in the corpus to generate a dictionary, and then convert the words into corresponding codes to initially achieve digital representation of the text.

[0064] Word embedding: The preprocessed text data is segmented at the character and word granularity to construct character embedding and word embedding.

[0065] S3, select three base classifiers, and adopt a serial training method for the three base classifiers. Each base classifier depends on the feature vector generated by the previous base classifier during the training process. Input the training data set, and train the training data set in sequence according to the three serial base classifiers. Each base classifier outputs the corresponding confidence.

[0066] Specifically, the three base classifiers are the TextRCNN model, the FastText model, and the Transformer model.

[0067] The present invention selects three deep learning classification models with good classification performance and complementary advantages, TextRCNN, FastText, and Transformer, as base classifiers, focusing on extracting local feature information, continuous word order information, and long-term dependency information of the text respectively. In order to enhance the relevance of the three base classification models and combine their respective advantages to increase the feature richness of the training text, a serial training method is adopted, and each base classifier depends on the feature vector generated by the previous model during the training process.

[0068] Combine the following Figure 2 This method selects the TextRCNN model ( Figure 2 model1 in), FastText model ( Figure 2 model2) and Transformer model ( Figure 2Model3 in is used as the base classifier. Among them, the TextRCNN model is used to extract local and global features of the text and fine-grained text information. The FastText model adds n-gram vectors to extract continuous word order information and uses a fully connected neural network structure to capture shallow information of the text. The Transformer model adds word position vectors to capture positional relationships and uses the self-attention mechanism to better solve long-term dependency problems. These three classification methods have shown relatively good performance in text classification. Selecting these three diverse and complementary models as base classifiers can more effectively extract complementary information at different levels, thereby further improving the classification accuracy of the integrated model.

[0069] This method adopts a serial training method in the model training part. Each model depends on the feature vector generated by the previous model during the training process. Specifically, a serial training method is adopted to add the penultimate feature vector of the fully connected layer of the TextRCNN model to the input of the fully connected layer of the FastText model, and the penultimate feature vector of the fully connected layer of the FastText model is added to the input of the linear layer of the Transformer model.

[0070] The training data set is input, and the training data set is trained in sequence according to the three base classifiers connected in series as mentioned above, and each base classifier outputs the corresponding confidence level.

[0071] S4, taking the confidence output by the three base classifiers as input data, searches for weights suitable for each base classifier on the validation data set, and uses the particle swarm optimization (PSO) algorithm to calculate and adjust the weights, performs weight adjustment with feedback, determines the optimal weight of each base classifier through iterative feedback, and uses the optimal weight to perform weighted fusion on the corresponding confidences to obtain the fused confidence.

[0072] Specifically, calculating the confidence after fusion includes the following steps:

[0073] (1) Initialization: Assume there are k base classifiers, and initialize the weight vector W with length k = {w1,w2,…,w k}.

[0074] The present invention adopts three base classifiers, that is, there are three base classifiers in total, and the weight vector W={w1, w2, w3} with an initialization length of 3.

[0075] (2) Update: The PSO algorithm is used to adjust and optimize the weight vector through iterative calculation to find the optimal weight.

[0076] Specifically, the weight vector is adjusted and optimized through iterative calculation to find the optimal solution. In each iteration, the particle is updated according to the individual extreme value (pbest) and the global extreme value (gbest). The individual extreme value is the best position experienced by the particle itself, and the global extreme value is the best position experienced by the entire particle group. The calculation of the particle speed and position iteration process is shown in formula (1):

[0077]

[0078] Among them, t represents the number of iterations, v i and p i They represent the speed and position of the i-th particle respectively. α is the inertia coefficient. The larger the value, the less likely it is to change the previous movement route. c1 is the individual acceleration factor. The larger the value, the more likely the particle is to choose the best position it has experienced. c2 is the social acceleration factor. The larger the value, the more likely the particle is to choose the best position experienced by the entire particle group. r 1i and r 2i They represent the randomness of the i-th particle in the direction of seeking individual extrema and global extrema, and are random numbers in the range of [0,1]. gbest is taken as the final result at the end of the iteration.

[0079] (3) Feedback: In the iterative process, the fitness function is used as a criterion for measuring the quality of a particle's position. In the present invention, the fitness function uses binary cross entropy loss.

[0080]

[0081] Among them, fitness is the adaptation function, l ij represents the true value of the i-th sample label j, s ij represents the probability of label j on sample i, N is the number of samples, and Z is the number of labels.

[0082] The PSO algorithm optimizes the particle position by continuously updating and feedback operations, and finally finds the global optimal value gbest as the weight vector of model fusion, determines the optimal weight of each base classifier, and performs weighted fusion according to the optimal weight of each base classifier and the confidence of the output to obtain the confidence after fusion.

[0083] The confidence after fusion is calculated using the following formula:

[0084]

[0085] Among them, Score is the confidence of the entire integrated model after model fusion, k is the number of base classifiers, k = 3, w iIndicates the weight corresponding to each base classifier obtained by the PSO algorithm, Score i =[s i1 ,…,s iZ ] represents the confidence of the output of the i-th base classifier, and Z is the number of labels.

[0086] S5, obtaining the classification result of the text data according to the fused confidence, and taking the category with the highest confidence among the multiple fused confidences obtained as the predicted classification result.

[0087] Specifically, the fused confidence has multiple values. Confidence refers to the set of probabilities that the sample belongs to each category, so the confidence contains multiple probability values. In this step, the category with the largest probability value is taken as the predicted classification category. That is, among the multiple fused confidences obtained, the category with the highest confidence is taken as the predicted classification result.

[0088] Then, the accuracy of the classification results is tested using the test data set. Specifically, the data in the test data set is input into the text classification model to obtain the classification accuracy, which is repeated more than three times to obtain the average value to verify the performance of the classification model.

[0089] In summary, the present invention first selects three deep learning models, TextRCNN, FastText, and Transformer, as base classifiers, and then trains each base classifier in a serial training mode to obtain three sets of classification confidence. Finally, the PSO algorithm is used to adjust the weights with feedback on the validation set to obtain the optimal weights suitable for each base classifier, thereby obtaining the weighted final classification category prediction, thereby improving the robustness and classification accuracy of the text classification model.

[0090] The text classification method based on ensemble learning of the present invention can achieve higher accuracy classification of text data, and is applicable to the intelligent system operation and maintenance functions of the visual networking platform in the fields of telecommunications, etc. As the premise and basis for realizing intelligent operation and maintenance, operation and maintenance data classification has strong application demand and practical significance, and provides strong support for operation and maintenance fault analysis and solution generation.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A text classification method based on ensemble learning, characterized in that: include: S1, collect samples and mark categories, build a data set, and divide the data set into a training data set, a validation data set, and a test data set in a preset ratio; S2, preprocessing the data of the training data set, the validation data set and the test data set, including performing text standardization, text cleaning, word segmentation, word cleaning and normalization, and establishing a dictionary and word embedding on the text; S3, selecting three base classifiers, adopting a serial training method for the three base classifiers, each base classifier depends on the feature vector generated by the previous base classifier during the training process; inputting a training data set, training the training data set in sequence according to the three serially connected base classifiers, and each of the base classifiers outputs a corresponding confidence level; S4, using the confidences output by the three base classifiers as input data, searching for weights suitable for each base classifier on the validation data set, and using a particle swarm optimization (PSO) algorithm to calculate and adjust the weights, and determining the optimal weight of each base classifier through iterative feedback, and using the optimal weights to perform weighted fusion on the corresponding confidences to obtain fused confidences; S5, obtaining the classification result of the text data according to the fused confidence, and taking the category with the highest confidence among the multiple fused confidences obtained as the predicted classification result.

2. The text classification method based on ensemble learning according to claim 1, characterized in that: In S1, the dataset is divided into a training dataset, a validation dataset, and a test dataset in a ratio of 6:2:

2.

3. The text classification method based on ensemble learning according to claim 1, characterized in that: In S2, the data in this article is cleaned and normalized, including: removing stop words, part-of-speech tagging, word form restoration, spelling correction, extracting valid words, and restoring the word part of speech and tense; The dictionary is established for the data in this paper, including: firstly, the words appearing in the corpus are counted and coded to generate a dictionary, and then the words are converted into corresponding codes to preliminarily realize the digital representation of the text; The data in this article is embedded in words, including: segmenting the data after text preprocessing at the character and word granularity, and constructing character embedding and word embedding.

4. The text classification method based on ensemble learning according to claim 1, characterized in that: In S3, the three base classifiers are respectively a TextRCNN model, a FastText model, and a Transformer model.

5. The text classification method based on ensemble learning according to claim 4, characterized in that: A serial training method is adopted, where the penultimate feature vector of the fully connected layer of the TextRCNN model is added to the input of the fully connected layer of the FastText model, and the penultimate feature vector of the fully connected layer of the FastText model is added to the input of the linear layer of the Transformer model.

6. The text classification method based on ensemble learning according to claim 4, characterized in that: The TextRCNN model is used to extract local and global features of text and fine-grained text information; The FastText model is used to add n-gram vectors to extract continuous word order information, and uses a fully connected neural network structure to capture shallow information of the text; The Transformer model is used to add position vectors of words to capture positional relationships.

7. The text classification method based on ensemble learning according to claim 1, characterized in that: In S4, the confidence after fusion is calculated, including: (1) Assume there are three base classifiers, and initialize the weight vector W = {w1, w2, w3} with a length of 3; (2) The PSO algorithm is used to adjust and optimize the weight vector through iterative calculation to find the optimal weight; (3) In the iterative process, the fitness function is used as a criterion for measuring the quality of a particle's position. The PSO algorithm optimizes the particle position by continuously updating and feedback operations, and finally finds the global optimal value gbest as the weight vector for model fusion, determines the optimal weight of each base classifier, and performs weighted fusion based on the optimal weight of each base classifier and the output confidence to obtain the fused confidence.

8. The text classification method based on ensemble learning according to claim 7, characterized in that: The adaptation function uses binary cross entropy loss, Among them, fitness is the adaptation function, l ij represents the true value of the i-th sample label j, s ij represents the probability of label j on sample i, N is the number of samples, and Z is the number of labels.

9. The text classification method based on ensemble learning according to claim 8, characterized in that: The confidence after fusion is calculated using the following formula: Among them, Score is the confidence of the entire integrated model after model fusion, k is the number of base classifiers, k = 3, w i Represents the weight of each base classifier obtained by the PSO algorithm, Score i =[s i1 ,…,s iZ ] represents the confidence of the output of the i-th base classifier, and Z is the number of labels.

10. The text classification method based on ensemble learning according to claim 1, characterized in that: After S5, the method further includes: using the test data set to test the accuracy of the classification result.

Images