Intelligent comment information classification method fusing long text theme keywords
By combining technologies such as BERT, UMAP, HDBSCAN, ProdLDA, GloVe and multi-layer neural network, short text sparsity and noise problems in social network comments are solved, and high consistency theme mining and high-precision negative comment detection are achieved, improving Emotional feature extraction ability.
Patent Information
- Application Number
- CN202411225192.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-29
AI Technical Summary
In the analysis of social network comment topics, due to the sparseness and noise of short texts, it is difficult for the existing technology to achieve high consistency theme mining and high-precision negative comment detection, and the deep learning model lacks the ability to extract deep emotional features.
The BERT model is used to extract topic words vectors and comment word vectors, combine UMAP and HDBSCAN algorithms for dimensionality reduction and clustering, build a core word set of comment topics, use ProdLDA model for topic extraction, and use GloVe and pre-trained BERT model for word embedding, combined with Multi-layer neural networks perform local feature extraction and context information capture, and ultimately use the emotion classification model to achieve binary classification detection.
It realizes highly consistent review text theme mining and high-precision negative comment detection, which improves the ability to extract deep emotional features and alleviates the sparseness and noise problems of short texts.
Smart Images

Figure CN119046466A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent processing of comment information, and involves text feature enhancement technology, information theme extraction technology, short text aggregation technology and text sentiment analysis technology. It is an intelligent classification method for comment information integrating long text theme keywords. Technical Background
[0002] For the analysis of social network comment topics, due to the short length of social network opinion texts, the data is sparse and contains a lot of noise, resulting in poor results in the analysis of opinion topics. In the current research on short text topic modeling, most scholars mainly focus on the sparsity problem of short texts. For the method of forming pseudo-long documents by aggregation, when short texts with different semantic contents are aggregated into long documents, the co-occurrence of non-related words will produce incoherent topics. For the method of global document word co-occurrence, only the information of words in the document set is paid attention to, ignoring the internal structure of the document, such as word order and semantic structure. For the method of assuming that each short text contains only one topic, although the sparsity problem of short texts is alleviated to a certain extent, it also makes the model unable to capture the multiple topics of the document. For the sentiment analysis of social network comments, due to the irregularity of social network texts, short texts have a lot of implicit semantics, which requires high text feature extraction. At present, the sentiment classification method based on deep learning shows excellent performance and can achieve preliminary semantic understanding, but the ability to extract deep sentiment features in texts needs to be further improved. Summary of the invention
[0003] The purpose of the present invention is to provide an intelligent classification method for comment information that integrates long text topic keywords to address the above-mentioned shortcomings, integrate deep learning technology, and propose an optimized model algorithm under specific scenario tasks, so as to achieve highly consistent comment text topic mining and high-precision negative comment detection.
[0004] The present invention is achieved by adopting the following technical solutions: S1. Use the BERT model to extract the topic word vector and comment word vector of the comment information text, and use the topic attention layer to enhance the comment information text features; S2, use the UMAP algorithm to reduce the dimension of the comment vector, and use the HDBSCAN algorithm to cluster the comment text; S3, construct the core word set of the comment topic, aggregate the short text of the comment to obtain the long comment document; S4, using ProdLDA neural topic model to extract review topics; S5. Convert the review text into two different types of word embeddings using the GloVe model and the pre-trained BERT model. S6. Use a multi-layer neural network model to extract local features and capture contextual information from text; S7. Use the sentiment classification model to implement binary classification detection on the comment text.
[0005] The specific process of step S1 includes the following steps: S1-1, extract the topic field in the comment information text, and add a "[SEP]" mark after the topic field in the comment text; S1-2. Input the marked comment text into the BERT model, learn the text representation through the multi-layer Transformer architecture, and obtain the last hidden layer output; S1-3. According to the added tags, the topic word vector matrix and the comment word vector matrix are obtained respectively, and the attention matrix is generated to obtain the similarity between each comment word and the topic word; S1-4, using the attention matrix and the comment word vector matrix obtained in step S1-3 to generate a weighted comment word vector matrix; S1-5. Convert the weighted comment word vector matrix into sentence vector representation through the pooling layer.
[0006] The specific process of step S2 includes the following steps: S2-1. Use the UMAP algorithm, a nonlinear dimensionality reduction method based on manifold learning, to reduce the dimensionality of the comment vector, preserving the local and global topological structure of high-dimensional data in low-dimensional space; S2-2. Use the density-based clustering method HDBSCAN algorithm to perform cluster analysis on the comment text after dimensionality reduction to obtain a cluster set.
[0007] The specific process of step S3 includes the following steps: S3-1. Using the TF-IDF algorithm, for each word in each cluster, the weight of each word in the cluster is calculated according to the number of times it appears in the cluster and the frequency of its appearance in the entire public opinion dataset; S3-2, according to the weight of each word in the cluster, select several words with the highest weight from each cluster as core words, form a core word set of the comment topic, and record the weight of each word; S3-3. For each cluster set, according to a specific aggregation strategy, the short texts in the cluster are aggregated by a certain amount to form long texts.
[0008] The specific process of step S4 includes the following steps: S4-1. Build the ProdLDA model, use the aggregated long review document as the model input, and use the constructed core word set as the decoder reconstruction target; S4-2, use Adam optimizer and batch normalization and other techniques to train the encoder network to convert document feature representation into potential topic distribution representation; S4-3. After the model training is completed, the encoder network is used to convert the newly input review text data into a potential topic distribution representation.
[0009] The specific process of step S5 includes the following steps: S5-1. Use Chinese word segmentation tools (such as Jieba library) to segment and tag the preprocessed text, and encode the marked parts of speech; S5-2. For each word in the text, calculate the position code according to its position index and dimension index; S5-3. Input the comment text, pass it through the word embedding layer, fuse the part-of-speech encoding obtained in step S5-1 and the position encoding obtained in step 5-2, and obtain the GloVe word embedding sequence and the BERT word embedding sequence.
[0010] The specific process of step S6 includes the following steps: S6-1, use the Glove word embedding sequence obtained in step S5-3 as the input of the neural network CNN-MHSA layer, extract the local features of the comment text through convolution operation, and obtain the feature vector output after pooling operation, multi-head attention operation and feature fusion. ; S6-2, use the BERT word embedding sequence obtained in step S5-3 as the input of the BiGRU-ATT layer of the neural network, use two independent GRU networks to process the forward and reverse sequence data respectively, add an attention layer before the output layer of the BiGRU model, and use the weighted sum feature vector output by the attention layer as the final output feature vector of the BiGRU-ATT layer .
[0011] The specific process of step S7 includes the following steps: S7-1, concatenate the two feature vectors obtained in S6-1 and S6-2 to obtain a fused feature vector; S7-2, input the fused feature vector into the fully connected layer for feature transformation and dimensionality reduction; S7-3. Use the output of the fully connected layer as the input of the Softmax classification layer to obtain the classification result. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 It is a processing flow diagram of the model of the present invention; Figure 2 This is a schematic diagram of the overall framework of the comment sentiment analysis model. DETAILED DESCRIPTION
[0013] See attached Figure 1 This embodiment describes a method for intelligently classifying review information that integrates long text topic keywords as follows: (1) The BERT model is used to extract topic word vectors and comment information word vectors of the comment information text, and the topic attention layer is used to enhance the features of the comment information text.
[0014] (1-1) Extract the topic field in the comment information text and add a "[SEP]" tag after the topic field in the comment text to identify the location of the topic field.
[0015] (1-2) Input the marked comment text into the BERT model, learn the text representation through the multi-layer Transformer architecture, and obtain the last hidden layer output containing the vector representation of each word.
[0016] (1-3) According to the added tags, we obtain the topic word vector matrix A and the comment word vector matrix B respectively, using the formula M = AB T Generate the attention matrix M and obtain the similarity between each comment word and the topic word.
[0017] (1-4) Use the Softmax function along the column dimension of the attention matrix M to normalize the similarity between the comment word vector and the topic word, and then multiply it by the word vector. Perform the above calculation for each comment word vector to obtain the weighted comment word vector matrix.
[0018] (1-5) The weighted comment word vector matrix is converted into a sentence vector representation through the pooling layer. The average pooling strategy is adopted here, and the average value of all word vectors is used as the comment sentence vector. For comments that do not contain topic fields, the word vector matrix is obtained by inputting the BERT model and then directly entering the pooling layer.
[0019] (2) The UMAP algorithm is used to reduce the dimension of the comment vector, and the HDBSCAN algorithm is used to cluster the comment text.
[0020] (2-1) The UMAP algorithm is used to reduce the dimensionality of the comment vector. First, the distance between each point in the data set and its nearest neighbor is calculated, and then a local neighborhood graph is constructed. This process involves using the k-nearest neighbor algorithm to identify the neighbors of each point and adjusting the weights through Gaussian distribution to accurately reflect the local distance between the points. Furthermore, the high-dimensional comment vector is mapped into a low-dimensional space representation, which ensures the structural integrity of the data points by minimizing the cross entropy of the neighborhood graphs in high-dimensional and low-dimensional spaces.
[0021] (2-2) The HDBSCAN algorithm is used to cluster the comment texts. First, a minimum spanning tree of data points is constructed. The edge weights of the tree are based on the mutual reach distance between sample points. Then, the minimum spanning tree is converted into a clustering tree structure. This process involves sorting the edges in the tree in ascending order of distance, and merging the two subgraphs connected by the edges in turn to form hierarchical clusters. The tree is then pruned based on a density threshold to form clusters.
[0022] (3) Construct a core word set of comment topics and aggregate short comment texts to obtain long comment documents.
[0023] (3-1) Using the TF-IDF algorithm, for each word in each cluster, the weight of each word in the cluster obtained in step (2-2) is calculated based on the number of times it appears in the cluster and the frequency of its appearance in the entire public opinion dataset. The weight is positively correlated with the number of times the word appears in the cluster and negatively correlated with the frequency of its appearance in the entire dataset.
[0024] (3-2) Based on the obtained word weights, several words with higher weights are further selected from each cluster as core words to complete the construction of the core word set of the comment topic, where each word has a weight to indicate its importance.
[0025] (3-3) For each cluster set, according to a specific aggregation strategy, the short texts in the cluster are aggregated into long texts according to a certain number (such as every 10). For the remaining short texts in the cluster that do not meet the aggregation number, they are aggregated into the last long text unit. The co-occurrence and context information of the words in the text are increased through aggregation, which alleviates the sparsity problem of short texts in topic modeling.
[0026] (4) Use the ProdLDA neural topic model to extract review topics.
[0027] (4-1) Initialize the ProdLDA model, set the model parameters, including the number of network layers, number of nodes, learning rate, etc., use the aggregated public opinion long document as the model input, and use the constructed core word set as the decoder reconstruction target.
[0028] (4-2) The encoder network is trained using techniques such as the Adam optimizer and batch normalization to convert document feature representations into potential topic distribution representations. The decoder network samples from the potential representations and attempts to map them back to the original document data, iterating and optimizing until the model converges or reaches the preset number of training rounds.
[0029] (4-3) After the model training is completed, the encoder network is used to convert the newly input public opinion data into a potential topic distribution representation. Based on the topic distribution representation, the topic category to which each document belongs is determined.
[0030] (5) The review text is converted into two different types of word embeddings through the GloVe model and the pre-trained BERT model.
[0031] (5-1) Use Chinese word segmentation tools (such as Jieba library) to segment and tag the preprocessed text, which includes identifying the part-of-speech category of each word in the text, such as noun, verb, adjective, etc., and encoding the marked part-of-speech to ensure that the dimension of each part-of-speech vector is consistent with the dimension of the GloVe word vector.
[0032] (5-2) For each word in the text, the position encoding is calculated based on its position index and dimension index to enhance the model's understanding of long-distance dependencies.
[0033] (5-3) For the input comment text, the part-of-speech embedding obtained in step (5-1) and the position encoding obtained in step S5-2 are fused with the GloVe word embedding to enhance the model's understanding of the text semantics, and a fused GloVe word embedding sequence is obtained; the comment text is processed using the pre-trained BERT model, and the output is obtained from the last hidden layer of the BERT model to obtain a BERT word embedding sequence.
[0034] (6) Use a multi-layer neural network model to extract local features and capture contextual information from text.
[0035] (6-1) The Glove word embedding sequence obtained in step (5-3) is used as the input of the neural network CNN-MHSA layer. First, the input sequence is convolved using convolution kernels with a height of 2, 3, and 4. The number of convolution kernels of each height is set to 128, and the width of each convolution kernel is consistent with the dimension of the input word vector. The local features of the public opinion text are extracted through the convolution operation; then, the output of each convolution kernel is subjected to a maximum pooling operation, and the window size is set to 3 to extract the most significant features from each convolution feature map; further, the feature vector obtained by the pooling layer is linearly transformed, and a multi-head attention mechanism is introduced to calculate the attention weight for the output of each channel; finally, the outputs of the three local feature extraction channels are fused to obtain the final output feature vector of the CNN-MHSA layer. .
[0036] (6-2) Use the BERT word embedding sequence obtained in step (5-3) as the input of the BiGRU-ATT layer of the neural network, use two independent GRU networks to process the forward and reverse sequence data respectively, merge the forward and reverse hidden states at each time point, and obtain the feature vector at each time point. Add an attention layer before the output layer of the BiGRU model, calculate the similarity with the input vector to update the weights of each dimension, and use the weighted sum feature vector output by the attention layer as the final output feature vector of the BiGRU-ATT layer. .
[0037] (7) Use the sentiment classification model to perform binary classification detection on the comment text.
[0038] (7-1) Concatenate the two feature vectors obtained by (6-1) and (6-2) to obtain a fused feature vector.
[0039] (7-2) Input the fused feature vector into the fully connected layer for feature transformation and dimensionality reduction.
[0040] (7-3) The output of the fully connected layer is used as the input of the Softmax classification layer to obtain the classification result. Here, binary classification is used, and the cross-entropy loss function is used to calculate the prediction loss. In supervised learning, the cross-entropy loss function is used to evaluate the difference between the label distribution predicted by the model and the true label distribution, and to guide model training.
Claims
1. An intelligent classification method for review information integrating long text theme keywords, It is characterized in that The steps include: S1. Use the BERT model to extract the topic word vector and comment word vector of the comment information text, and use the topic attention layer to enhance the comment information text features; S2, use the UMAP algorithm to reduce the dimension of the comment vector, and use the HDBSCAN algorithm to cluster the comment text; S3, construct the core word set of the comment topic, aggregate the short text of the comment to obtain the long comment document; S4, using ProdLDA neural topic model to extract review topics; S5. Convert the review text into two different types of word embeddings using the GloVe model and the pre-trained BERT model. S6. Use a multi-layer neural network model to extract local features and capture contextual information from text; S7. Use the sentiment classification model to implement binary classification detection on the comment text.
2. The intelligent classification method for review information integrating long text theme keywords according to claim 1, It is characterized in that The specific analysis process of step S1 includes the following steps: S1-1, extract the topic field in the comment information text, and add a "[SEP]" mark after the topic field in the comment text; S1-2. Input the marked comment text into the BERT model, learn the text representation through the multi-layer Transformer architecture, and obtain the last hidden layer output; S1-3. According to the added tags, the topic word vector matrix and the comment word vector matrix are obtained respectively, and the attention matrix is generated to obtain the similarity between each comment word and the topic word; S1-4, using the attention matrix and the comment word vector matrix obtained in step S1-3 to generate a weighted comment word vector matrix; S1-5. Convert the weighted comment word vector matrix into sentence vector representation through the pooling layer.
3. The intelligent classification method for review information integrating long text subject keywords according to claim 1, It is characterized in that The specific analysis process of step S2 includes the following steps: S2-1. Use the UMAP algorithm, a nonlinear dimensionality reduction method based on manifold learning, to reduce the dimensionality of the comment vector, preserving the local and global topological structure of high-dimensional data in low-dimensional space; S2-2. Use the density-based clustering method HDBSCAN algorithm to perform cluster analysis on the comment text after dimensionality reduction to obtain a cluster set.
4. The intelligent classification method for review information integrating long text subject keywords according to claim 1, It is characterized in that The specific analysis process of step S3 includes the following steps: S3-1. Using the TF-IDF algorithm, for each word in each cluster, the weight of each word in the cluster is calculated according to the number of times it appears in the cluster and the frequency of its appearance in the entire public opinion dataset; S3-2, according to the weight of each word in the cluster, select several words with the highest weight from each cluster as core words, form a core word set of the comment topic, and record the weight of each word; S3-3. For each cluster set, according to a specific aggregation strategy, the short texts in the cluster are aggregated by a certain amount to form long texts.
5. The intelligent classification method for review information integrating long text subject keywords according to claim 1, It is characterized in that The specific analysis process of step S4 includes the following steps: S4-1. Build the ProdLDA model, use the aggregated long review document as the model input, and use the constructed core word set as the decoder reconstruction target; S4-2, use Adam optimizer and batch normalization and other techniques to train the encoder network to convert document feature representation into potential topic distribution representation; S4-3. After the model training is completed, the encoder network is used to convert the newly input review text data into a potential topic distribution representation.
6. The intelligent classification method for review information integrating long text subject keywords according to claim 1, It is characterized in that The specific analysis process of step S5 includes the following steps: S5-1. Use Chinese word segmentation tools (such as Jieba library) to segment and tag the preprocessed text, and encode the marked parts of speech; S5-2. For each word in the text, calculate the position code according to its position index and dimension index; S5-3. Input the comment text, pass it through the word embedding layer, fuse the part-of-speech encoding obtained in step S5-1 and the position encoding obtained in step 5-2, and obtain the GloVe word embedding sequence and the BERT word embedding sequence.
7. The intelligent classification method for review information integrating long text subject keywords according to claim 1, It is characterized in that The specific analysis process of step S6 includes the following steps: S6-1, use the Glove word embedding sequence obtained in step S5-3 as the input of the neural network CNN-MHSA layer, extract the local features of the comment text through convolution operation, and obtain the feature vector output after pooling operation, multi-head attention operation and feature fusion. ; S6-2, use the BERT word embedding sequence obtained in step S5-3 as the input of the BiGRU-ATT layer of the neural network, use two independent GRU networks to process the forward and reverse sequence data respectively, add an attention layer before the output layer of the BiGRU model, and use the weighted sum feature vector output by the attention layer as the final output feature vector of the BiGRU-ATT layer .
8. The intelligent classification method for review information integrating long text subject keywords according to claim 1, It is characterized in that The specific analysis process of step S7 includes the following steps: S7-1, concatenate the two feature vectors obtained in S6-1 and S6-2 to obtain a fused feature vector; S7-2, input the fused feature vector into the fully connected layer for feature transformation and dimensionality reduction; S7-3. Use the output of the fully connected layer as the input of the Softmax classification layer to obtain the classification result.
Citation Information
Cited By
Ceramic censer design method and system based on perceptual engineering
CN120562241A
Ceramic incense burner design method and system based on kansei engineering
CN120562241B
Live broadcast comment analysis method and device, storage medium and computer equipment
CN120632100A
Topic model generation method and device, storage medium and program product
CN121117220A
