A sentiment feature extraction method based on BI-LSTM-CNN with orthogonal projection
By orthogonal projection of neutral word vectors in the BI-LSTM-CNN model and combining the CNN-BI-LSTM-Attention mechanism, the shortcomings of traditional models in capturing high-distinguishing emotional characteristics are solved, and the emotional classification of text is improved. Accuracy and feature integrity.
Patent Information
- Application Number
- CN202210167102.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-23
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-02-23
AI Technical Summary
Traditional RNN and CNN deep learning models are difficult to capture the complete emotional characteristics with high discrimination in text emotion classification, especially ignoring the characters and local context information of special meanings, resulting in the inadequate richness and accuracy of emotional characteristics.
Using the BI-LSTM-CNN method based on orthogonal projection, the neutral word vector is projected into the emotional polar word vector space, and the appropriate weight is given to special meaning words, and combined with the CNN-BI-LSTM-Attention model, the high discrimination of the Emotional characteristics.
It improves the accuracy and discrimination of text emotion classification, enhances the integrity of emotional characteristics and the generalization performance of the model, and provides stronger technical support for the construction of emotional thesaurus and text emotion classification.
Smart Images
Figure CN114528407B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing, and in particular to a sentiment feature extraction method of a BI-LSTM-CNN based on orthogonal projection. Background Art
[0002] Text sentiment classification is the process of analyzing, processing, summarizing and reasoning subjective texts with emotional connotations. It is also a computational study of people's opinions, emotions, evaluations and attitudes towards products, services, events and their attributes. It is a subtask of text classification. Different from ordinary text classification, text sentiment classification requires higher-level semantic extraction and highly discriminative sentiment features, which is more technically challenging. Therefore, how to use natural language processing (NLP) technology to classify subjective opinion texts is attracting more and more researchers' attention.
[0003] Traditional RNN and CNN deep learning models have made significant progress in generating discriminative features through their powerful representation learning capabilities, but the completeness of sentiment features still needs to be improved. For example, both RNN and CNN learning models can be used for text sentiment classification. Among them, the RNN model can model the entire sentence and capture the long-term dependencies within the sentence; however, modeling the entire sequence may ignore some key local contextual information that is crucial for classification. The CNN model can extract more local and position-invariant features through windows of different sizes, but it cannot assign appropriate weights to certain special characters (for example, words with special meanings such as "general", "normal", "?", etc.). Therefore, it is necessary to capture the long-term dependencies within the sentence while assigning appropriate weights to certain special characters to obtain complete sentiment features with high discrimination, which is one of the current research hotspots.
[0004] To solve this problem, an effective sentiment feature extraction method should consider the following aspects: (1) How to obtain accurate sentiment polarity word vector matrices and characters with special meanings from text; (2) How to weight certain special words or characters to obtain more discriminative sentiment features; (3) How to improve the accuracy of text sentiment classification by extracting sentiment feature vectors from higher-level semantics. Summary of the invention
[0005] Based on the above considerations, the present invention introduces a method of neutral word vector projection, which can assign appropriate weights to some words with special meanings to obtain highly discriminative sentiment features. A sentiment feature extraction method based on BI-LSTM+CNN (CPD-NET+CBA) based on orthogonal projection is proposed. The method is divided into two parts, namely the orthogonal projection model (CPD-NET) and the complex neural network model (CNN+BI-LSTM+Attention, CBA). The core task of the model is to project the neutral word vector into the orthogonal space, assign it appropriate weights, and then learn sentiment features with higher discrimination through the CBA model, so that the text is more discriminative in sentiment classification.
[0006] The method comprises the following steps:
[0007] 1) A text sequence is converted into a wordvector sequence as the input of the model through a word embedding model such as Word2Vec.
[0008] 2) Calculate the semantic association information between words, set appropriate thresholds, and filter out neutral word vectors or symbols with special meanings.
[0009] 3) Project the selected neutral word vectors or symbols into the sentiment polarity word vector space, assign them appropriate weights, and obtain weighted neutral word vectors.
[0010] 4) Use the CNN-BI-LSTM hybrid neural network to extract the key semantics of the text and perform reinforcement learning to fully mine the text information and obtain the text sentiment feature vector.
[0011] 5) Introduce the Attention mechanism to calculate the matching score between the output at each moment and the entire feature vector, and assign appropriate weights to each feature vector.
[0012] 6) The sentiment word vector extracted by the complex neural network and feature projection network obtains the cross-information between features through the fully connected layer, and learns sentiment features with higher discrimination, making the sentiment classification more discriminative.
[0013] The advantages and positive effects of the present invention are as follows: the present invention discloses a sentiment feature extraction method based on BI-LSTM-CNN of orthogonal projection to improve the accuracy of text sentiment classification. The present invention obtains highly discriminative sentiment features through orthogonal space and deep learning with more discriminative semantics, increases the integrity of sentiment feature extraction, and makes the text more discriminative in sentiment classification. At the same time, the sentiment word library constructed by this method can provide a method reference for the construction of other word libraries (for example: the construction of sentiment word libraries for product reviews, the construction of sentiment word libraries for microblog topics, etc.) to improve the accuracy of text sentiment classification. In addition, the method combines the advantages of deep learning networks, performs reinforcement learning, fully mines the sentiment features of text, improves the classification accuracy and generalization performance of the model, and provides technical support for text sentiment classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 This is a flowchart of a sentiment feature extraction method based on BI-LSTM-CNN with orthogonal projection;
[0015] Figure 2 It is an orthogonal projection (CPD-NET) model of the sentiment feature extraction method based on BI-LSTM-CNN of orthogonal projection;
[0016] Figure 3 It is a word correlation graph of the sentiment feature extraction method of BI-LSTM-CNN based on orthogonal projection;
[0017] Figure 4 This is a complex network structure framework diagram of a BI-LSTM-CNN sentiment feature extraction method based on orthogonal projection;
[0018] Figure 5 This is a multi-model example diagram of a sentiment feature extraction method based on BI-LSTM-CNN with orthogonal projection; DETAILED DESCRIPTION
[0019] The present invention will be further described below:
[0020] Combination Figure 1, the specific implementation method is as follows: The present invention discloses a sentiment feature extraction method based on BI-LSTM-CNN with orthogonal projection, which aims to obtain weighted neutral word vectors from text, obtain sentiment features with higher discrimination, and provide strong technical support for text sentiment classification, etc. Traditional deep learning models ignore words with special meanings in key local context information, resulting in insufficiently rich sentiment features. To address this problem, the method proposed in the present invention projects the neutral word vector into the orthogonal space of sentiment polarity words to obtain a weighted neutral word vector, and extracts key semantics of the text through a CNN deep learning model; in addition, the BI-LSTM-Attention model and the weighted neutral word vector are used to learn from the extracted key semantics to enhance the semantic features of sentence sentiment, making the text more discriminative in sentiment classification.
[0021] Combination Figure 2 The specific implementation method is as follows: the method proposed in the present invention will give appropriate weights to some words with special meanings, that is, by calculating the relevance of the neutral word context, the selected neutral word vectors are projected into the sentiment polarity word vector space, and the neutral word vectors are given appropriate weights. Figure 2 As shown. The whole model consists of two parts, namely the feature projection network (CP-NET) and the sentiment polarity feature learning network (D-NET). The model projects the learned text complete information vector into a more discriminative semantic space to increase the influence of special neutral word vectors on text sentiment classification. The purpose of CP-NET is to extract common target features, namely neutral word feature vectors, and assign appropriate weights; the purpose of D-NET is to obtain sentiment polarity word vectors. CP-NET consists of three parts: input layer X, feature extractor F cp , orthogonal projection layer (OPL); D-NET consists of input layer X, feature extractor F d (Feature Extractor F cp and feature extractor F d The core idea of this model is to transform the feature extractor F cp The calculated eigenvector f cp Projected to feature extractor f d The extracted feature vector f d In the orthogonal direction, that is, f cp Projecting into a space with more discriminative semantics allows some special neutral words to be given appropriate weights, increasing the completeness of sentiment feature extraction.
[0022] Combination Figure 3, the specific implementation method is: the method proposed in the present invention uses weighted neutral word vectors to represent the importance of certain special words, that is, when setting labels for texts, there will generally be features with clearer emotional features, such as: perfect and nice represent positive emotions, and bad and disgusting represent negative emotions. However, there will also be words in the text that have other obvious features, called neutral words (for example: "general", "normal", "?", etc.). Their emotional features are not obvious, and there is no ambiguity in classification. They all belong to neutral word vectors, but neutral word vectors will affect the text sentiment classification results to a certain extent.
[0023] Specifically, the Degree of Word Correlation (DWC) is used to measure the correlation between sentiment polarity words and neutral words in the context, that is, to calculate the sentiment polarity strength of neutral words from the context. With the help of the semantic correlation between words, words or characters with special meanings can be more accurately identified, thereby improving the completeness of feature extraction. The calculation formula is as follows: i =|C i ×WC i |As shown. Among them, C i It represents the correlation between neutral words and polar words. Its calculation formula is as follows: As shown. u represents the average value in the entire review corpus, R u,i Indicates the correlation between word i and the average value, R u,j WC represents the correlation between word j and the average value. i Indicates the part of speech of a neutral word, WC i By W i and L i The calculation formula is as follows: i =W i ×L i As shown. Where W i represents the initial weight of neutral words, L i Indicates the position of a word. Since there are many neutral words in the text, and only a few neutral words have special meanings, the LTP tool of Harbin Institute of Technology is used to segment the text according to the part of speech, calculate the semantic association between words, and filter out neutral words or symbols with special meanings. Figure 3 As shown in the figure, "computer", "general", and "!" are all neutral words. By calculating the semantic association between neutral words and polar words, words greater than the threshold are selected as screening results, providing data support for subsequent feature projection operations.
[0024] Combination Figure 4, the specific implementation method is as follows: the method proposed in the present invention combines the neural network model with the attention mechanism, and proposes a CBA (CNN+BI-LSTM+Attention) model. The CNN+BI-LSTM hybrid neural network is used to extract the emotional features of the input text, and the attention mechanism is combined to obtain rich emotional features. First, the model uses the Word2vec model to convert the input text into a word vector and pass it into the CNN-BI-LSTM hybrid neural network; secondly, the input vector is fully mined by taking advantage of the CNN model to obtain the main features of the input text. The CNN model includes a convolution layer and a maximum pooling layer. The purpose of the convolution layer is to obtain the features of the input data, and the activation functions of both layers are Relu. The maximum pooling operation is performed after the convolution layer, and the features extracted by the convolution layer are downsampled to achieve the purpose of streamlining the model parameters; then, the features extracted by the CNN network are passed into the BI-LSTM network, and the text features are extracted for reinforcement learning to obtain the intrinsic connection between the features to solve the long-term dependency problem; finally, the Attention mechanism is introduced to calculate the matching score between the output at each moment and the entire feature vector. The larger the score, the greater the corresponding attention, and the more important the word is. This method mainly includes embedding layer, CNN-BI-LSTM layer and Attention layer.
[0025] Combination Figure 5 , the specific implementation method is: in the iterative process of multi-model comparison, the parameter optimization route of the method disclosed in the present invention is more efficient. In the early stage of training (the first 10 rounds), the accuracy of each model is low and the cross entropy loss is high, but the accuracy of the method disclosed in the present invention is in a leading position; in the middle stage of training (10 to 20 rounds), the cross entropy loss function is used to calculate the loss value and optimize the parameters of the model, the accuracy of other multiple models gradually improves, but the accuracy of the method disclosed in the present invention begins to stabilize; in the late stage of training (20 to 50 rounds), the accuracy of other multiple models gradually stabilizes.
[0026] The curves of the traditional CNN network and LSTM network fluctuate greatly and begin to converge at the fifth Epoch. The curves of the improved hybrid model of CNN and LSTM are relatively flat and fluctuate less, but the convergence speed is still relatively slow. The method disclosed in the present invention not only has a faster convergence speed, but also a relatively flat overall curve, a stable training process and a high accuracy rate, and has always been in a leading position in the subsequent training process, further reflecting the advantages of the method disclosed in the present invention.
[0027] In addition, the above embodiments are only used to illustrate the specific embodiments of the present invention rather than to limit it. Those skilled in the art should understand that some of the technical features therein may be replaced by equivalents, and these modifications and replacements also fall within the protection scope of the present invention.
Claims
1. A sentiment feature extraction method based on BI-LSTM-CNN with orthogonal projection aims to obtain weighted neutral word vectors from texts, obtain sentiment features with higher discrimination, and provide strong technical support for text sentiment classification; Different from traditional deep learning models, this method can capture special meaning words in key local context information and obtain richer sentiment features; It is characterized by: The neutral word vector is projected into the orthogonal space of sentiment polarity words to obtain the weighted neutral word vector. At the same time, the key semantics of the text are extracted through the CNN deep learning model. The BILSTM-Attention model and the weighted neutral word vector are used to learn the semantic features that can enhance the sentiment of the sentence from the extracted key semantics, making the text more discriminative in sentiment classification. The emotional feature extraction method specifically includes: The neutral word feature vector is extracted through the feature projection network CP-NET and assigned appropriate weights; the sentiment polarity word vector is obtained through the sentiment polarity feature learning network D-NET; The CP-NET includes an input layer X, a feature extractor F cp , orthogonal projection layer; the D-NET includes an input layer X, a feature extractor F d , gradient reverse layer; the feature extractor F cp The calculated eigenvector f cp Projection to feature extractor f d The extracted feature vector f d In the orthogonal direction, that is, f cp Projecting into a more discriminative semantic space, so that some special neutral words are given appropriate weights; The sentiment polarity strength of neutral words is calculated from the context, and the semantic associations between words are used to accurately identify words or characters with special meanings, thereby improving the completeness of feature extraction.
2. The method according to claim 1, characterized in that The neutral word feature vector is projected onto the sentiment polarity word feature vector, and appropriate weights are assigned to the results obtained based on the projection; the sentiment of the text is classified using the feature vector obtained through the projection.
3. The method according to claim 1, characterized in that For the CNN deep learning model, by utilizing its internal weight sharing and local connection structure, CNN can effectively extract deep features contained in the data while reducing the complexity of the algorithm; the convolutional neural network consists of a convolutional layer that performs convolution calculations on the data and extracts potential features, and a pooling layer that downsamples and compresses the network parameters; the convolutional layer and the pooling layer are used alternately; the convolutional neural network is used to first extract key semantics from the text, and then pass it into the BI-LSTM network for prediction.
4. The method according to claim 1, characterized in that The entire sentence is modeled to capture the long-term dependencies within the sentence. When modeling the entire sequence, some key local contextual information that is crucial to classification can be captured. Then, the Attention mechanism is used. In the field of natural language processing, the larger the weight of a word, the more important it is in the entire text, and the greater the role it plays in the entire classification task. Focusing on words that have a greater impact on the text classification results can effectively improve the classification effect.