A text classification method based on fusion features and improved LSTM

Abstract

This invention discloses a text classification method based on fused features and an improved LSTM, comprising: acquiring text data and dividing it into training and test sets; preprocessing the text to obtain cleaned text data; extracting text features using the Word2Vec method to obtain static feature vectors; extracting text features using a pre-trained BERT Chinese model to obtain context-inclusive feature vectors; fusing the static feature vectors and context-inclusive feature vectors to obtain fused features; inputting the fused features from the training set into an improved LSTM network for model training; and using the trained classification model to perform classification validation on the test set to evaluate the model's performance. This method aims to combine static and dynamic feature vectors, leveraging the advantages of both, and optimizing the weight allocation of input features through an attention mechanism, thereby strengthening the model's focus on key information. This invention is applicable to various fields requiring high-precision text classification.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a text classification method based on fused features and improved LSTM. Background Technology

[0002] With the rapid development of the internet and information technology, massive amounts of text data, such as news reports, social media posts, product reviews, and legal documents, are constantly being generated. Text classification, as a fundamental task in natural language processing, is widely used in various information processing systems, such as news classification, sentiment analysis, spam detection, and public opinion monitoring.

[0003] In recent years, the field of text classification has seen rapid technological advancements, particularly the introduction of deep learning models, which has significantly improved the accuracy and efficiency of text classification. Models based on various neural networks can effectively process the sequential information of text and capture the contextual relationships within it. Furthermore, the emergence of models pre-trained using massive amounts of unsupervised data has further enhanced the performance of text classification models.

[0004] However, existing models still face challenges in effectively capturing long-range dependencies and key information when processing long texts. Text feature extraction often neglects semantics or loses context; most feature extraction models are singular and cannot simultaneously consider both static and dynamic features. In traditional neural networks, all input information is processed equally, but in text classification, different words or phrases have varying impacts on the final classification. Furthermore, increasingly deep neural networks face the problem of vanishing or exploding gradients, making training extremely difficult. Error information from deep networks is difficult to propagate effectively back to earlier layers, making it hard for models to capture complex patterns. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention discloses a text classification method based on fused features and an improved LSTM. In feature extraction, it takes into account both static features and contextual content, and introduces an attention mechanism into the existing LSTM network to increase the weight of key information in the calculation process, so that the network focuses on important information. At the same time, residual connections are used to ensure the fluidity of information and avoid the degradation problem of the model.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a text classification method based on fused features and improved LSTM, comprising the following steps:

[0007] S1. Obtain text data and divide it into training and test sets;

[0008] S2. Preprocess the text to obtain cleaned text data;

[0009] S3. Use the Word2Vec method to extract the features of the text and obtain a static feature vector;

[0010] S4. Use the pre-trained BERT Chinese model to extract features from the text and obtain a feature vector containing the context.

[0011] S5. Merge the static feature vector and the feature vector containing the context to obtain the fused feature;

[0012] S6. Input the fused features of the training set into the improved LSTM network for model training;

[0013] S7. Use the trained classification model to perform classification validation on the test set to evaluate the model's performance.

[0014] Preferably, in step S2, the preprocessing steps for the text data include:

[0015] Remove HTML tags, special characters, and extra spaces; convert all letters in the text to lowercase; remove stop words.

[0016] Preferably, in step S3, the Word2Vec feature vector is obtained by training the Word2Vec model. The Word2Vec model is trained using text data, and the specific steps include:

[0017] Using the cleaned text data, a Word2Vec model is trained to generate word embedding vectors. The Word2Vec model employs the CBOW method, training each word based on a context window to obtain a 768-dimensional vector representation for each word. The steps include:

[0018] By utilizing the vocabulary in the email text, a context window is constructed, selecting a certain range of context words as input and the target word as output;

[0019] The context word vectors are averaged to obtain the context feature vector h;

[0020] The CBOW model is used to predict target words using context vector h, and the score of each word is probabilistically processed using the Softmax function. The specific formula of the Softmax function is as follows:

[0021]

[0022] Among them, v ωt Indicates target word ω t The word vectors are given by P(ω), where h is the average vector of the surrounding words, V is the vocabulary, and P(ω) is the word vector. t |ω t-n ,…,ωt-n ) represents the target word ω given the context. t The predicted probability.

[0023] Preferably, in step S4, when extracting text features using the pre-trained BERT Chinese model, the specific steps include:

[0024] The email text is segmented using a pre-trained BERT word segmenter to obtain a word sequence, which is then passed as input to the BERT model.

[0025] The word sequence is encoded using a pre-trained BERT Chinese model to obtain a context-related representation for each word. The context-related representation is a dynamic word vector, in which the BERT model adjusts the semantic representation of each word according to the occurrence of the word in different contexts.

[0026] The feature vector of each word is extracted from the encoding result output by the BERT model. The feature vector is then used to obtain the global feature representation of the text through the CLS tag of the last layer. This global feature representation is a 768-dimensional vector and includes contextual information.

[0027] Preferably, in step S5, the static feature vector extracted by Word2Vec and the context feature vector extracted by BERT are fused. The fusion step includes:

[0028] Feature concatenation: The feature vectors extracted by the Word2Vec and BERT models are directly concatenated to form a 1536-dimensional feature representation;

[0029] Feature Dimension Adjustment: To avoid computational overhead due to excessively high dimensionality, Principal Component Analysis (PCA) is further used to reduce the dimensionality of the fused feature vector, resulting in a 768-dimensional feature vector. PCA includes the following steps:

[0030] 1) Standardize the fused feature vectors;

[0031] 2) Calculate the covariance matrix of the standardized feature matrix;

[0032] 3) Perform eigenvalue decomposition on the covariance matrix and select the first 768 principal components;

[0033] 4) Project the standardized feature vectors onto the 768 principal components to obtain the dimensionality-reduced fused feature vectors.

[0034] Preferably, in step S6, the improved LSTM network includes both an attention mechanism and residual connections. The attention mechanism dynamically calculates the attention weight of each input based on the current input information and the hidden state information of the previous time step, and optimizes the input of the LSTM by weighted fusion of the input information. The residual connections are introduced in each layer of the LSTM, and directly sum the current input information with the output after LSTM transformation, thereby ensuring that the information flow is better transmitted in the network and effectively alleviating the gradient vanishing problem.

[0035] Preferably, the attention mechanism calculates the current input vector x. t The hidden state h from the previous moment t-1 and cell state c t-1 To generate attention weights a t , will the current input x t and the calculated attention weight a t Multiplying them together yields the optimized input vector x′. t The attention weights are input into the LSTM unit, and the formula is as follows:

[0036] a t =σ a (W a x t +U a h t-1 +M a c t-1 +b a )

[0037] Weighted input formula:

[0038] x′ t =a t ·x t

[0039] Where σ a It is the Sigmoid activation function, W a U a M a Let b be the weight matrix of the attention mechanism. a For bias, a t This represents the attention weight at the current moment.

[0040] Preferably, the residual connections of the LSTM unit are implemented in the following way:

[0041] For the input vector x' at time t t The output after LSTM transformation is F(x') t The residual connection is achieved by connecting the input x' t With output F(x') t Adding them together gives the final output y.t :

[0042] y t =F(x') t )+x' t

[0043] F(x' t ) = h t

[0044] Output gate:

[0045] o t =σ(W O x' t +U O h t-1 +b0)

[0046] h t =o t ·tanh(c t )

[0047] Cell state:

[0048]

[0049] Candidate cell status:

[0050]

[0051] Input Gate:

[0052] i t =σ(W i x' t +U i h t-1 +b i )

[0053] Forgotten Gate:

[0054] f t =σ(W f x' t +U f h t-1 +b f )

[0055] Among them, h t c is the current hidden state. t For the current cell state, W O W c W i W f U represents the weight matrices for different gates. O U c U i U fThe hidden state h from the previous time step t-1 The effect on different gates, b0, b c b i b f For bias.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] 1. This invention improves the accuracy of text classification. By fusing static features extracted by Word2Vec and contextual features extracted by BERT, it can simultaneously consider the basic semantic information of words and the dynamic semantic changes in the context, thereby providing a more comprehensive and accurate feature representation for text classification models.

[0058] 2. This invention optimizes the performance of LSTM networks. By introducing an attention mechanism and residual connections into the LSTM network, this invention effectively alleviates the gradient vanishing problem that LSTM models may encounter when processing long texts. The attention mechanism enables the model to adaptively focus on important parts of the text, while residual connections ensure the effective transfer of information between each layer, thereby improving the stability and accuracy of the model in processing long sequences. Attached Figure Description

[0059] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0060] In the attached diagram:

[0061] Figure 1 This is a schematic diagram of a text classification method based on fused features and improved LSTM provided in an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of an LSTM unit according to an embodiment of the present invention;

[0063] Figure 3 This is a comparison test diagram of the verification accuracy of the improved LSTM network of this invention with LSTM, CNN, and RNN. Detailed Implementation

[0064] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0065] Example: Figures 1-3 As shown, a text classification method based on fused features and improved LSTM includes the following steps:

[0066] Step 1: Obtain text data and divide it into training and test sets: 75% of the text data is divided into the training set and the remaining 25% into the test set.

[0067] Step 2: Preprocess the text to obtain the cleaned text data:

[0068] (1) Remove HTML tags and special characters: When processing input text, first remove all HTML tags from the text (e.g., ...). , , etc.), as well as possible special characters (such as,, &, <, >, etc.). These HTML tags and special characters are irrelevant to the actual semantics of the text and may interfere with subsequent text analysis, so they need to be removed.

[0069] (2) Remove extra spaces: There may be extra spaces, tab characters, line break characters and other unnecessary characters in the text. These characters not only waste computing resources but may also interfere with subsequent text tokenization and feature extraction. Therefore, remove the extra spaces to ensure the standardization and consistency of the text.

[0070] (3) Convert all letters to lowercase: To avoid inconsistencies in vocabulary processing caused by case differences, convert all letters in the text to lowercase. This step helps to simplify text feature extraction and improve the model processing efficiency, avoiding redundant features due to different letter cases.

[0071] (4) Remove stop words: Stop words (such as "of", "is", "in", "and", etc.) usually do not contribute significantly to the understanding of the text in natural language processing but appear frequently in the text. By removing these stop words, the dimension of the text can be reduced, the amount of calculation can be decreased, and at the same time, the subsequent text representation can be more focused on meaningful content.

[0072] Step 3: Use the Word2Vec method to extract the features of the text and obtain static feature vectors;

[0073] Use the Word2Vec method based on the gensim library to process all the text data in the training set. Set the dimension of the word vector to 768, the size of the context window to 5, the minimum limit of the word frequency to 1, and use the CBOW method for training.

[0074] According to the vocabulary in the text, construct a context window, and select the context words within a certain range in the window as the input and the target word as the output for model training. [[ID= nineteen]]

[0075] Average the word vectors corresponding to the context words to obtain the feature vector h of the context as the context representation of the target word;

[0076] Predict the target word through the context vector h and probabilize the scores of each word through the Softmax function. Among them, the specific formula of the Softmax function is as follows:

[0077]

[0078] Among them, v ωt represents the word vector of the target word ω t h is the average vector of the context words, V is the vocabulary, P(ω t |ω t-n ,…,ω t-n ) represents the target word ω given the context. t The predicted probability,

[0079] Step 4: Use the pre-trained BERT Chinese model to extract features from the text and obtain a feature vector containing the context;

[0080] In this implementation scheme, the BERT model used is derived from the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". To adapt to Chinese text processing, the Chinese version based on this paper, bert-base-chinese, is used. This model has been specifically trained and optimized for Chinese text processing and can effectively capture contextual information in Chinese text.

[0081] (1) Use the pre-trained BERT word segmenter to segment the email text to obtain a word sequence, and pass the word sequence as input to the BERT model;

[0082] (2) The word sequence is encoded using a pre-trained BERT Chinese model to obtain a context-related representation of each word. The context-related representation is a dynamic word vector, wherein the BERT model can adjust the semantic representation of each word according to the occurrence of words in different contexts.

[0083] (3) Extract the feature vector of each word from the encoding result output by the BERT model. The feature vector is used to obtain the global feature representation of the text through the CLS tag of the last layer (which is a 768-dimensional vector). The global feature representation contains context information.

[0084] Step 5: Fuse the static feature vector and the feature vector containing the context to obtain the fused feature;

[0085] (1) Feature concatenation: The feature vectors extracted by Word2Vec and BERT models are directly concatenated to form a 1536-dimensional feature representation, as shown in the formula;

[0086] f combined =[f bert ,f w2v ]

[0087] Where f bert The 768-dimensional vector generated for BERT, f w2v f is a 768-dimensional vector generated by Word2Vec. combined This is the concatenated 1536-dimensional vector.

[0088] (2) Adjusting the Dimensionality of the Fusion Feature Vector: To avoid the computational overhead caused by high-dimensional features, PCA (Principal Component Analysis) is further used to reduce the dimensionality of the concatenated fusion feature vector. PCA can project high-dimensional feature vectors into a low-dimensional space while preserving as much of the main information of the data as possible. After PCA processing, the dimension of the fusion feature vector will be reduced to 768 to meet the needs of subsequent model training and computation.

[0089] The fused feature vectors are standardized so that each dimension follows a standard normal distribution with a mean of 0 and a variance of 1. The standardization formula is:

[0090]

[0091] Where f is the original feature vector, μ is the mean of the feature, and σ is the standard deviation of the feature.

[0092] For the standardized feature matrix X norm Calculate the covariance matrix. The covariance matrix reflects the linear relationship between features of different dimensions, and the formula is as follows:

[0093]

[0094] Where C is the covariance matrix, n is the number of samples, and X... norm It is the standardized feature matrix.

[0095] Eigenvalue decomposition is performed on the covariance matrix C to obtain eigenvalues ​​and corresponding eigenvectors. The first 768 principal components are selected as the new projection space. The formula for eigenvalue decomposition is:

[0096]

[0097] v i It is the i-th eigenvector, λ i These are the corresponding eigenvalues.

[0098] The standardized fusion feature vector is projected onto the selected first 768 principal components to obtain the dimensionality-reduced fusion feature vector. The projection formula is:

[0099] f pca =X norm V 768

[0100] Where V 768 It is the eigenvector matrix of the first 768 principal components, f pca It is the fused feature vector after dimensionality reduction.

[0101] Ultimately, the fused feature vector Fpca obtained through PCA dimensionality reduction has 768 dimensions, effectively reducing computational complexity while preserving the main semantic information.

[0102] Calculate the covariance matrix of the standardized feature matrix;

[0103] Eigenvalue decomposition of the covariance matrix was performed, and the first 768 principal components were selected.

[0104] The standardized feature vectors are projected onto the 768 principal components to obtain the dimensionality-reduced fused feature vectors.

[0105] Step 6: Input the fused features of the training set into the improved LSTM network for model training;

[0106] The improved LSTM network introduces an attention mechanism and residual connections to ensure that important information is not lost;

[0107] Depend on Figure 1 As can be seen, the attention mechanism calculates the current input vector x. t The hidden state h from the previous moment t-1 and cell state c t-1 To generate attention weights a t , will the current input x t and the calculated attention weight a t Multiplying them together yields the optimized input vector x'. t The attention weights are input into the LSTM unit, and the formula is as follows:

[0108] a t =σ a (W a x t +U a h t-1 +M a c t-1 +b a )

[0109] Weighted input formula:

[0110] x' t =a t ·x t

[0111] Where σ a It is the Sigmoid activation function, W a U a M a Let b be the weight matrix of the attention mechanism. a For bias, a t This represents the attention weight at the current moment.

[0112] The residual connections of the LSTM unit are implemented in the following way: for the input vector x′ at time t... t The output after LSTM transformation is F(x′) t The residual connection is achieved by connecting the input x' t With output F(x') t Adding them together gives the final output y. t :

[0113] y t =F(x′) t )+x' t

[0114] F(x′ t ) = h t

[0115] Output gate:

[0116] o t =σ(W O x' t +U O h t-1 +b0)

[0117] h t =o t ·tanh(c t )

[0118] Cell state:

[0119]

[0120] Candidate cell status:

[0121]

[0122] Input Gate:

[0123] i t =σ(W i x' t +U i h t-1 +b i )

[0124] Forgotten Gate:

[0125] f t =σ(W f x' t +U f h t-1 +b f )

[0126] Among them, h t c is the current hidden state. t For the current cell state, W O W c W i W f U represents the weight matrices for different gates. O U c U i U f The hidden state h from the previous time step t-1 The effect on different gates, b0, b c b i b f For bias.

[0127] By introducing attention mechanisms and residual connections into the LSTM network, the gradient vanishing problem that LSTM models may encounter when processing long texts is effectively alleviated. The attention mechanism enables the model to adaptively focus on important parts of the text, while residual connections ensure the effective transfer of information between each layer, thereby improving the stability and accuracy of the model in processing long sequences.

[0128] Step 7: Use the trained classification model to perform classification verification on the test set.

[0129] To verify the effectiveness of the improved LSTM, it was compared with other neural network models such as CNN on the Chinese dataset THUCNews.

[0130] LSTM and RNN were compared, and the models were trained for 10, 30, and 50 epochs respectively. The accuracy of the models was compared, and the experimental results are shown in Table 1.

[0131]

[0132]

[0133] As shown in Table 1, when using different neural network models in the experiment, the improved LSTM achieved the best accuracy. It improved the accuracy by 2.01% compared to LSTM after 10 training iterations, by 1.16% after 30 training iterations, and by 1.08% after 50 training iterations.

[0134] The accuracy of RNN and LSTM varies considerably with different training iterations, while the improved LSTM model shows similar training results. The accuracy difference between training 10 times and training 50 times is only 0.13%, indicating that the improved LSTM has good generalization ability while having high accuracy. Moreover, the model is not sensitive to changes in the number of training iterations and can stably adapt to different data and tasks.

[0135] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A text classification method based on fused features and improved LSTM, characterized in that, Includes the following steps: S1. Obtain text data and divide it into training and test sets; S2. Preprocess the text to obtain cleaned text data; S3. Use the Word2Vec method to extract the features of the text and obtain a static feature vector; S4. Use the pre-trained BERT Chinese model to extract features from the text and obtain a feature vector containing the context. S5. Merge the static feature vector and the feature vector containing the context to obtain the fused feature; S6. Input the fused features of the training set into the improved LSTM network for model training; S7. Use the trained classification model to perform classification validation on the test set to evaluate the model's performance; In step S3, the Word2Vec feature vector is obtained by training the Word2Vec model. The Word2Vec model is trained using text data, and the specific steps include: Using the cleaned text data, a Word2Vec model is trained to generate word embedding vectors. The Word2Vec model employs the CBOW method, training each word based on a context window to obtain a 768-dimensional vector representation for each word. The steps include: By utilizing the vocabulary in the email text, a context window is constructed, selecting a certain range of context words as input and the target word as output; The context word vectors are averaged to obtain the context feature vector. ; Using the CBOW model, through context vectors Predict the target word, and then apply a probabilistic processing method to the score of each word using the Softmax function. The specific formula for the Softmax function is as follows: ； in, Indicate target word Word vectors, The average vector of the context words. For the vocabulary list, For target words under given context conditions The predicted probability; In step S4, when extracting text features using the pre-trained BERT Chinese model, the specific steps include: The email text is segmented using a pre-trained BERT word segmenter to obtain a word sequence, which is then passed as input to the BERT model. The word sequence is encoded using a pre-trained BERT Chinese model to obtain a context-related representation for each word. The context-related representation is a dynamic word vector, in which the BERT model adjusts the semantic representation of each word according to the occurrence of the word in different contexts. The feature vector of each word is extracted from the encoding result output by the BERT model. The feature vector is then used to obtain the global feature representation of the text through the CLS tag of the last layer. This global feature representation is a 768-dimensional vector and includes contextual information. In step S6, the improved LSTM network includes both an attention mechanism and residual connections. The attention mechanism dynamically calculates the attention weight of each input based on the current input information and the hidden state information of the previous time step, and optimizes the input of the LSTM by weighted fusion of the input information. The residual connections are introduced in each layer of the LSTM and directly sum the current input information with the output after LSTM transformation. The attention mechanism calculates the input vector at the current time. The state that was hidden a moment ago and cell state To generate attention weights , will the current input and the calculated attention weights Multiply to obtain the optimized input vector. The attention weights are input into the LSTM unit, and the formula is as follows: ； Weighted input formula: ； in It is the Sigmod activation function. , Here is the weight matrix for the attention mechanism. For bias, The attention weight at the current moment; The residual connections of the LSTM units are implemented in the following way: For the Input vector at time step The output after LSTM transformation is Residual connections are achieved by connecting the input... With output Add them together to get the final output. : ； ； Output gate: ； ； Cell state: ； Candidate cell status: ； Input Gate: ； Forgotten Gate: ； in, The current hidden state, The current cell state, , , , For the weight matrices of different gates, , , , The hidden state of the previous moment The impact on different doors , , , For bias.

2. The text classification method based on fusion features and improved LSTM according to claim 1, characterized in that: In step S2, the preprocessing steps for the text data include: Remove HTML tags, special characters, and extra spaces; convert all letters in the text to lowercase; remove stop words.

3. The text classification method based on fusion features and improved LSTM according to claim 1, characterized in that: In step S5, the static feature vector extracted by Word2Vec and the context feature vector extracted by BERT are fused. The fusion steps include: Feature concatenation: The feature vectors extracted by the Word2Vec and BERT models are directly concatenated to form a 1536-dimensional feature representation; Feature Dimension Adjustment: To avoid computational overhead due to excessively high dimensionality, Principal Component Analysis (PCA) is further used to reduce the dimensionality of the fused feature vector, resulting in a 768-dimensional feature vector. PCA includes the following steps: 1) Standardize the fused feature vectors; 2) Calculate the covariance matrix of the standardized feature matrix; 3) Perform eigenvalue decomposition on the covariance matrix and select the first 768 principal components; 4) Project the standardized feature vectors onto the 768 principal components to obtain the dimensionality-reduced fused feature vectors.

Images