Multi-level semantic analysis method based on fast text classification and BERT

Abstract

The invention provides a multilevel semantic analysis method based on fast text classification and BERT, and belongs to the technical field of natural language process.The method comprises the steps that a text data set is obtained, data in the text data set is preprocessed, and the preprocessed text data set is obtained; constructing a multi-level semantic analysis model by utilizing fast text classification, BERT, feature fusion and sentiment classification; training a multi-level semantic analysis model by using the preprocessed text data set to obtain an optimized multi-level semantic analysis model; performing multi-level semantic analysis on the preprocessed text data by using the optimized multi-level semantic analysis model to obtain a multi-level semantic analysis result; the problems that in the prior art, the low-frequency word and unlisted word processing capacity is poor, effective fusion of multi-level semantics is lacked, and the sentiment analysis accuracy is low are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a multi-level semantic analysis method based on fast text classification and BERT. Background Art

[0002] With the continuous development of artificial intelligence technology, sentiment analysis has become an important application in the field of natural language processing (NLP), especially in the fields of education, medical care, finance, etc. In the field of online education, automated sentiment analysis can not only help analyze students' feedback on courses and teachers' teaching effects, but also provide valuable decision support for education platforms.

[0003] At present, the mainstream methods of sentiment analysis can be divided into the following categories:

[0004] 1. Traditional text representation methods: such as Bag-of-Words and TF-IDF (Term Frequency-Inverse Document Frequency). These methods count the word frequencies of the text, but ignore the order relationship between words and the semantic information of the context. Therefore, they are not ideal for processing polysemous words and sentences with complex emotions.

[0005] 2. Sentiment analysis methods based on deep learning: such as RNN (recurrent neural network) and LSTM (long short-term memory network). These methods can capture the temporal information of text through recursive structure and gradually deepen the understanding of context, but their disadvantage is that the training time is long, especially when processing long texts, the gradient is prone to vanishing or exploding.

[0006] 3. Methods based on pre-trained models: In recent years, pre-trained language models such as BERT (Bidirectional Encoder based on Transformer) have been widely used in sentiment analysis tasks; BERT can better capture contextual information through bidirectional encoding and has achieved significant performance improvements; however, BERT still has certain shortcomings when dealing with low-frequency words and unregistered words (such as spelling errors and new words), because BERT relies on a lot of contextual information to understand vocabulary and performs poorly on low-frequency words.

[0007] Therefore, the main problems faced by existing sentiment analysis technologies include:

[0008] 1. The ability to process low-frequency words and unregistered words is weak, especially when facing new words or spelling errors, the accuracy is poor;

[0009] 2. It is difficult to capture global semantics and fine-grained contextual information at the same time, resulting in insufficient understanding of complex emotional expressions. Summary of the invention

[0010] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-level semantic analysis method based on fast text classification and BERT, which solves the problems of the prior art's weak processing ability for low-frequency words and unregistered words, lack of effective fusion of multi-level semantics, and low accuracy of sentiment analysis.

[0011] In order to achieve the above objectives, the technical solution adopted by the present invention is: a multi-level semantic analysis method based on fast text classification and BERT, comprising the following steps:

[0012] S1, obtaining a text data set, and preprocessing the data in the text data set to obtain a preprocessed text data set;

[0013] S2. Build a multi-level semantic analysis model using fast text classification, BERT, feature fusion, and sentiment classification;

[0014] S3, using the preprocessed text data set to train the multi-level semantic analysis model to obtain an optimized multi-level semantic analysis model;

[0015] S4. Use the optimized multi-level semantic analysis model to perform multi-level semantic analysis on the preprocessed text data to obtain a multi-level semantic analysis result.

[0016] The beneficial effects of the present invention are as follows: the present invention realizes accurate analysis of multi-level semantics by constructing a multi-level semantic analysis model based on fast text classification and BERT, improves the ability to process low-frequency words and unregistered words, realizes effective fusion of multi-level semantics, and improves the accuracy of sentiment analysis.

[0017] Furthermore, the S1 comprises the following steps:

[0018] S101, obtaining a text data set, and performing data cleaning and standardization on the data in the text set;

[0019] S102, using the Jieba library, decomposing the standardized text data into words or phrases;

[0020] S103, denoising the decomposed text data, and marking the sentiment tendency to obtain preprocessed text data, and organizing the preprocessed text data to obtain a preprocessed text data set.

[0021] The beneficial effect of the above further scheme is that the present invention ensures the clarity and standardization of input text data by preprocessing the text data set, removes irrelevant information, and improves the efficiency and accuracy of subsequent analysis.

[0022] Furthermore, the S2 comprises the following steps:

[0023] S201, using fast text classification, processing the vocabulary in the preprocessed text data set, and training the word vector by maximizing the probability of the word under the context vocabulary condition, to obtain the fast text classification word vector;

[0024] S202, performing vector representation on the words in the preprocessed text data set, and processing each vector using BERT to obtain a contextual representation of the text;

[0025] S203, using a method of feature fusion and concatenation weighted sum, the fast text classification word vector and the context representation of the text are fused to obtain a fused feature;

[0026] S204: Use the sentiment classifier to perform sentiment classification on the fused features, output the sentiment label, obtain the sentiment classification result, and complete the construction of a multi-level semantic analysis model based on fast text classification and BERT.

[0027] Furthermore, the expression of the feature fusion and concatenation weighted sum is as follows:

[0028] V fused =concat(V FastTest ,V Bert )

[0029] V fused =αV FastTest +βV Bert

[0030] Among them, V fused represents the fusion feature, V FastText Represents word vector, V Bert It represents the context representation of the text, and α and β both represent weight coefficients.

[0031] Furthermore, the expression of the sentiment classification is as follows:

[0032]

[0033] Among them, L(w,b) represents a classifier with w and b as parameters, y i represents the sentiment label of sample i, represents the fusion feature of sample i, and N represents the total number of samples.

[0034] The beneficial effects of the above further scheme are as follows: the present invention integrates the features of fast text classification and BERT extraction, thereby enhancing the accuracy and robustness of sentiment analysis; through the advantage of fast text classification not relying on a large amount of data, it has the ability to process relatively rare or non-appearing words in the training set; through BERT capturing contextual information, it understands the meaning of words from the global context, has the ability to process long texts, complex sentences and recognition of emotional expressions, accurately captures the emotional information in the text, enables the method of the present invention to better understand the true tendency of emotions, and reduces the misjudgment rate.

[0035] Furthermore, the S3 comprises the following steps:

[0036] S301, dividing the preprocessed text data set into a plurality of subsets according to the preprocessed text data set;

[0037] S302, according to each divided subset, calculating the error between the sentiment classification result in the subset and the true label, and obtaining a classification loss function;

[0038] S303, according to the classification loss function, using an optimization algorithm to update the parameters of the multi-level semantic analysis model, and using back propagation and gradient descent methods to adjust the multi-level semantic analysis model;

[0039] S304, recalculating the sentiment classification result and the classification loss function according to the updated multi-level semantic analysis model;

[0040] S305 . In response to the convergence of the classification loss function, an optimized multi-level semantic analysis model is obtained.

[0041] The beneficial effect of the above further scheme is: the present invention adopts cross-validation and parameter tuning methods to optimize the multi-level semantic analysis model, improves the stability of the multi-level semantic analysis model in processing various types of text data, and improves the generalization ability of the multi-level semantic analysis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is a flow chart of the method of the present invention.

[0043] Figure 2 Flow chart of sentiment analysis in the embodiment.

[0044] Figure 3 BERT structure diagram in the embodiment. DETAILED DESCRIPTION

[0045] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0046] Before describing this embodiment, the following terms are explained:

[0047] BERT: Transformer-based bidirectional encoder;

[0048] Jieba library: Jieba library, a third-party Chinese word segmentation library in Python;

[0049] FastTest: fast text classification;

[0050] n-gram: an algorithm based on statistical language models;

[0051] WordPiece: BERT’s method for splitting words into fragments.

[0052] Example

[0053] like Figure 1 As shown, the present invention provides a multi-level semantic analysis method based on fast text classification and BERT, and its implementation method is as follows:

[0054] S1. Obtain a text data set, and preprocess the data in the text data set to obtain a preprocessed text data set. The specific steps are as follows:

[0055] S101, obtaining a text data set, and performing data cleaning and standardization on the data in the text set;

[0056] S102, using the Jieba library, decomposing the standardized text data into words or phrases;

[0057] S103, denoising the decomposed text data, and marking the sentiment tendency to obtain preprocessed text data, and collating the preprocessed text data to obtain a preprocessed text data set.

[0058] In this embodiment, a text data set is collected (the data set may come from Chinese university MOOCs and product reviews, etc.), and the text data set is preprocessed. The preprocessing specifically includes:

[0059] Data cleaning, remove irrelevant information (such as HTML tags, stop words, and punctuation marks, etc.), and standardize the text data; word segmentation, use the Jieba library to decompose the standardized text data into words or phrases, so that the text data is suitable for a multi-level semantic analysis model; denoising and annotation, denoise the decomposed text data, remove noise words or meaningless words (such as irrelevant punctuation marks and special characters, etc.), and annotate the sentiment tendency (such as "positive", "negative" and "neutral") to obtain preprocessed text data, organize the preprocessed text data, and obtain a preprocessed text dataset.

[0060] S2. Use fast text classification, BERT, feature fusion, and sentiment classification to build a multi-level semantic analysis model. The specific steps are as follows:

[0061] S201, using fast text classification, processing the vocabulary in the preprocessed text data set, and training the word vector by maximizing the probability of the word under the context vocabulary condition, to obtain the fast text classification word vector;

[0062] S202, performing vector representation on the words in the preprocessed text data set, and processing each vector using BERT to obtain a contextual representation of the text;

[0063] S203, using a method of feature fusion and concatenation weighted sum, the fast text classification word vector and the context representation of the text are fused to obtain a fused feature;

[0064] S204: Use the sentiment classifier to perform sentiment classification on the fused features, output the sentiment label, obtain the sentiment classification result, and complete the construction of a multi-level semantic analysis model based on fast text classification and BERT.

[0065] In this embodiment, a multi-level semantic analysis model is constructed through fast text classification, BERT, feature fusion and sentiment classification;

[0066] Using FastText word vector training, a word and its context word set are obtained based on the preprocessed text dataset. The word vector is trained by maximizing the probability of the word under the context word condition. FastText can capture the fine-grained representation of each word and can effectively handle low-frequency words and unregistered words. The specific expression is as follows:

[0067]

[0068] Among them, P(c|w) represents the probability of maximizing the word under the condition of context vocabulary, w represents the vocabulary, c represents the context vocabulary, and v w The word vector v represents the word w. crepresents the word vector of the context word c, V represents all the words in the vocabulary, and v c′ Represents the word vector of the context word c′, where c′ represents a word in the vocabulary, and the FastText word vector is obtained by calculation;

[0069] According to the preprocessed text dataset, a sentence consisting of multiple words (or subwords) is obtained, where each word is represented by a vector and processed by the self-attention mechanism of the Transformer encoder in BERT. The specific expression is as follows:

[0070]

[0071] Where Attention(·) represents the self-attention mechanism, Q represents the query vector, K represents the key vector, V represents the value vector, softmax(·) represents the normalized exponential function, and d k Represents the dimension of the key vector; by matching the query of each word with the key of the context, the importance of each word in the context is obtained, and the contextual representation of the text is calculated;

[0072] The features extracted by FastText and BERT are fused, and the low-frequency word processing ability of FastText and the context understanding ability of BERT are used to obtain a comprehensive text representation; feature fusion is performed by concatenating and weighted summing, and optimized by cross-validation to obtain the best fusion effect and obtain fusion features; the expression of the feature fusion is as follows:

[0073] V fused =concat(V FastTest ,V BERT )

[0074] Among them, V fused represents the fusion feature, V FastText Represents word vector, V BERT Representing the context of the text;

[0075] The expression of the splicing weighted summation method is as follows:

[0076] V fused =αV FastTest +βV BERT

[0077] Among them, α and β both represent weight coefficients;

[0078] Use sentiment classifier (such as support vector product or neural network, etc.) to perform sentiment analysis on the fusion features; the sentiment classifier is a linear model, and the goal is set to minimize the classifier loss function. The sentiment classifier is pre-trained, and the sentiment classifier is used to extract sentiment information from the fusion features and perform accurate sentiment classification. The expression of the sentiment classification is as follows:

[0079]

[0080] Among them, L(w,b) represents a classifier with w and b as parameters, y i represents the sentiment label of sample i (y i =1 for the front side, y i = -1 for negative), represents the fusion features of sample i, and N represents the total number of samples; the sentiment classification result is obtained, and a multi-level semantic analysis model is constructed through fast text classification, BERT, feature fusion and sentiment classification.

[0081] S3. Use the preprocessed text dataset to train the multi-level semantic analysis model to obtain an optimized multi-level semantic analysis model. The specific steps are as follows:

[0082] S301, dividing the preprocessed text data set into a plurality of subsets according to the preprocessed text data set;

[0083] S302, according to each divided subset, calculating the error between the sentiment classification result in the subset and the true label, and obtaining a classification loss function;

[0084] S303, according to the classification loss function, using the optimization algorithm to update the parameters of the multi-level semantic analysis model, and using the back propagation and gradient descent method to adjust the multi-level semantic analysis model;

[0085] S304, recalculating the sentiment classification result and the classification loss function according to the updated multi-level semantic analysis model;

[0086] S305 . In response to the convergence of the classification loss function, an optimized multi-level semantic analysis model is obtained.

[0087] In this embodiment, the constructed multi-level semantic analysis model is optimized based on the preprocessed text data set, the preprocessed text data set is divided into multiple subsets, the multi-level semantic analysis model is trained and verified on different subsets using a cross-validation method, the error between the sentiment classification result and the true label is calculated, and a classification loss function (a commonly used loss function is a cross entropy loss) is obtained, an optimization algorithm (such as an Adam optimizer) is used to update the parameters of the multi-level semantic analysis model according to the classification loss function, and the multi-level semantic analysis model is adjusted using back propagation and gradient descent methods, and the sentiment classification result and the classification loss function are calculated again based on the updated multi-level semantic analysis model. When the classification loss function value converges, the updating of the multi-level semantic analysis model is stopped to obtain an optimized multi-level semantic analysis model.

[0088] S4. Use the optimized multi-level semantic analysis model to perform multi-level semantic analysis on the preprocessed text data to obtain a multi-level semantic analysis result.

[0089] In this embodiment, an optimized multi-level semantic analysis model is used to perform multi-level semantic analysis on the preprocessed text data to obtain a multi-level semantic analysis result, thereby completing a multi-level semantic analysis based on fast text classification and BERT.

[0090] In this embodiment, Figure 2 As shown in the figure, multi-level semantic analysis is performed, data is obtained through Python crawlers, and text data including student comments and student ratings are obtained from Chinese university MOOCs and product reviews; Python's Jieba library and large models are used for data preprocessing, meaningless comments and repeated comments are removed, and data annotation is performed with the assistance of large models to obtain analyzable and well-annotated comment data; FastText model and BERT model are used for data analysis, and the analyzable and well-annotated comment data are processed through WordPiece using the BERT model to obtain vocabulary, and the vocabulary is extracted to TokenEmbed dings (token embedding); use the FastText model to process the analyzable and well-annotated comment data through n-gram to obtain the FastText word vector, and determine whether the FastText word vector exists in the vocabulary. If so, extract the FastText word vector to the token embedding, otherwise, use n-gram to generate another word vector, and replace the FastText word vector with the additional word vector, and extract it to the token embedding; use TokenEmbeddings (token embedding) to fuse the vocabulary and word vector features, and perform sentiment analysis on the fused features to obtain the sentiment classification result, completing multi-level semantic analysis.

[0091] In this embodiment, Figure 3 As shown in the figure, a specific sentence "I like this course" is input, processed by BERT, and enters the input layer of BERT. The sentence is processed by the word segmenter, and combined with worldprice (which represents the word segmentation and ID mapping process of the input sentence by the word segmenter to help convert natural language into a numerical form that can be processed by the model) to get the id, and the id is input to the embedding layer for embedding. Through word embedding, position embedding and paragraph embedding, the embedding is added and normalized to get the embedding vector I, and the normalized embedding vectors I1, I2, ...IN are input into the Transformer model and processed by a single Transformer sublayer Tran in the Transformer model. Tran is mainly composed of a multi-head self-attention mechanism (Multi-Head Self-Attention) and a feed-forward network (Feed-Forward Network); residual connection and normalization are performed, and subsequent tasks are performed. Feature extraction is performed through the hidden layer, and finally the context representation of the text is output.

[0092] In this embodiment, when the comments contain complex emotions such as irony and humor, traditional sentiment analysis methods often cannot accurately identify them. The present invention can effectively improve the accuracy of sentiment classification by integrating the advantages of fast text classification and BERT.

Claims

1. A multi-level semantic analysis method based on fast text classification and BERT, characterized in that: The following steps are involved: S1, obtaining a text data set, and preprocessing the data in the text data set to obtain a preprocessed text data set; S2. Build a multi-level semantic analysis model using fast text classification, BERT, feature fusion, and sentiment classification; S3, using the preprocessed text data set to train the multi-level semantic analysis model to obtain an optimized multi-level semantic analysis model; S4. Use the optimized multi-level semantic analysis model to perform multi-level semantic analysis on the preprocessed text data to obtain a multi-level semantic analysis result.

2. The multi-level semantic analysis method based on fast text classification and BERT according to claim 1, characterized in that: The S1 comprises the following steps: S101, obtaining a text data set, and performing data cleaning and standardization on the data in the text set; S102, using the Jieba library, decomposing the standardized text data into words or phrases; S103, denoising the decomposed text data, and marking the sentiment tendency to obtain preprocessed text data, and collating the preprocessed text data to obtain a preprocessed text data set.

3. The multi-level semantic analysis method based on fast text classification and BERT according to claim 1, characterized in that: The S2 comprises the following steps: S201, using fast text classification, processing the vocabulary in the preprocessed text data set, and training the word vector by maximizing the probability of the word under the context vocabulary condition, to obtain the fast text classification word vector; S202, performing vector representation on the words in the preprocessed text data set, and processing each vector using BERT to obtain a contextual representation of the text; S203, using a method of feature fusion and concatenation weighted sum, the fast text classification word vector and the context representation of the text are fused to obtain a fused feature; S204: Use the sentiment classifier to perform sentiment classification on the fused features, output the sentiment label, obtain the sentiment classification result, and complete the construction of a multi-level semantic analysis model based on fast text classification and BERT.

4. The multi-level semantic analysis method based on fast text classification and BERT according to claim 3, characterized in that: The expressions of the feature fusion and concatenation weighted sum are as follows: In fused =concat(V FastTest ,V Bert ) V fused =αV FastTest +βV Bert Among them, V fused represents the fusion feature, V FastText Represents word vector, V Bert represents the context representation of the text, and α and β both represent weight coefficients.

5. The multi-level semantic analysis method based on fast text classification and BERT according to claim 3, characterized in that: The expression of the sentiment classification is as follows: Among them, L(w,b) represents a classifier with w and b as parameters, y i represents the sentiment label of sample i, represents the fusion feature of sample i, and N represents the total number of samples.

6. The multi-level semantic analysis method based on fast text classification and BERT according to claim 1, characterized in that: The S3 comprises the following steps: S301, dividing the preprocessed text data set into a plurality of subsets according to the preprocessed text data set; S302, according to each divided subset, calculating the error between the sentiment classification result in the subset and the true label, and obtaining a classification loss function; S303, according to the classification loss function, using the optimization algorithm to update the parameters of the multi-level semantic analysis model, and using the back propagation and gradient descent method to adjust the multi-level semantic analysis model; S304, recalculating the sentiment classification result and the classification loss function according to the updated multi-level semantic analysis model; S305 . In response to the convergence of the classification loss function, an optimized multi-level semantic analysis model is obtained.

Images