Text sentiment classification method and system based on adversarial training and improved BERT, electronic equipment and storage medium
By employing adversarial training and an improved BERT-based text sentiment classification method, the problems of lost sentiment information and insufficient semantic representation in both long and short texts are addressed. This approach enables effective modeling of complex semantics and transfer learning across text lengths, thereby improving the accuracy of sentiment classification.
Patent Information
- Application Number
- CN202511706564.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-02-17
AI Technical Summary
Existing methods suffer from problems such as loss of sentiment information, insufficient semantic representation, and difficulty in transfer learning across text lengths when processing texts of varying lengths, making it difficult to effectively model complex sentiment changes and negation relationships.
We employ an adversarial training and improved BERT text sentiment classification method. By combining text component recognition, knowledge graph enhancement, attention mechanism expansion, multi-scale feature fusion, and adaptive length processing with multimodal information, we generate deep semantic features and optimize them through adversarial training.
It achieves the integrity of sentiment information in both long and short texts and improves semantic representation. It can effectively capture the influence of complex semantics on sentiment tendency, effectively model dynamic sentiment changes, and make sentiment tendency judgment more accurate.
Smart Images
Figure CN121542432A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to a text sentiment classification method, system, electronic device, and storage medium based on adversarial training and an improved BERT. Background Technology
[0002] Text sentiment classification technology has evolved from traditional machine learning to deep learning, and then to pre-trained models. According to the latest research literature, the development of sentiment analysis technology can be divided into three main stages: traditional methods, deep learning methods, and pre-trained model methods. Traditional machine learning methods mainly include algorithms such as Support Vector Machines, Naive Bayes, Logistic Regression, and Random Forests. These methods treat sentiment analysis as a classification problem, achieving sentiment prediction through feature engineering and classification algorithms. Dictionary-based sentiment analysis is the most traditional and direct method, judging sentiment tendency by statistically analyzing the occurrence of sentiment words in the text. The emergence of deep learning methods has significantly improved the performance of sentiment classification. Recurrent Neural Networks and their variants can effectively capture sequence dependencies in text. Pre-trained model methods represent the current cutting edge of technology. Transformer models, through their self-attention mechanism, can effectively capture long-distance dependencies in text. BERT, as a bidirectional encoder representation model, considers the context on both sides of each word, significantly improving the understanding of complex sentence structures.
[0003] However, existing methods still suffer from the following problems: 1) Technical deficiencies in long and short text processing: Mainstream models such as BERT rely on fixed window truncation based on context, which leads to the loss of key sentiment information. Short texts suffer from insufficient semantic representation due to feature sparsity. 2) Difficulty in modeling sentiment changes: Long texts may contain multiple themes and sentiment changes. Existing models struggle to effectively model such complex dynamic sentiment changes and cannot effectively model complex semantics such as negation and contrast relationships. 3) Difficulty in cross-length text transfer learning: Short and long texts differ fundamentally in terms of context richness and lexical strength. Existing text classifiers struggle to achieve good generalization across text datasets of different lengths. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a text sentiment classification method, system, electronic device, and storage medium based on adversarial training and an improved BERT, which solves the problems of lost sentiment information in long texts, insufficient semantic representation in short texts, difficulty in modeling sentiment changes, and difficulty in transfer learning across text lengths in existing methods.
[0005] To achieve the above objectives, the present invention provides the following solution:
[0006] A text sentiment classification method based on adversarial training and an improved BERT includes:
[0007] The text to be analyzed is subjected to text component identification, format standardization, semantic decomposition and sentiment annotation, and contextual semantic association to obtain a preprocessed text sequence.
[0008] The pre-constructed knowledge graph is used to perform entity and relation matching, semantic feature supplementation, and vector transformation on the pre-processed text sequence to obtain an enhanced text vector sequence.
[0009] The enhanced text vector sequence is subjected to context expansion processing using an attention mechanism to obtain an expanded vector sequence.
[0010] When multimodal information exists, the extended vector sequence and the multimodal information are fused to obtain a multidimensional fused feature sequence;
[0011] The multi-dimensional fused feature sequence is subjected to adaptive sequence length processing based on a length threshold to obtain an enhanced text feature sequence;
[0012] The constructed text sentiment classification model is used to perform adaptive length encoding, positional encoding enhancement, multi-scale feature fusion, and attention fusion on the enhanced text feature sequence to obtain the text sentiment classification result; the training process of the text sentiment classification model includes:
[0013] Pre-collect target research texts and annotate the target research texts to obtain real text sentiment tags;
[0014] Extract the enhanced text feature sequence from the training set;
[0015] The enhanced text feature sequence is input into the improved BERT network for encoding to obtain deep semantic features;
[0016] Adversarial examples are generated based on the deep semantic features and the real text sentiment tags, and the improved BERT network is trained and optimized using the adversarial examples to obtain the text sentiment classification model.
[0017] Preferably, the text to be analyzed is subjected to text component identification, format standardization, semantic deconstruction and sentiment annotation, and contextual semantic association processing to obtain a preprocessed text sequence, including:
[0018] The text to be analyzed is analyzed using a text component recognition model to identify the components and obtain the component recognition results.
[0019] The format of the component identification results is corrected to obtain the corrected component representation;
[0020] Sentiment tags are applied to the sentiment words in the correction component representation based on a pre-constructed sentiment dictionary to obtain sentiment values; the expression for the sentiment values is:
[0021] ;in, The emotional value; For emotional words polarity; Intensity of emotion; For emotional words A set of negative words within a domain; express Elements within;
[0022] The logical relationships of the text to be analyzed are identified using dependency parsing, and the sentiment values of the text to be analyzed are weighted and calculated based on the logical relationships to obtain the preprocessed text sequence.
[0023] Preferably, the preprocessed text sequence is subjected to entity and relation matching, semantic feature supplementation, and vector transformation using a pre-constructed knowledge graph to obtain an enhanced text vector sequence, including:
[0024] Construct the knowledge graph; the knowledge graph includes: entities, attributes, and relationships;
[0025] The preprocessed text sequence is matched with the knowledge graph to obtain associated information;
[0026] The preprocessed text sequence is supplemented with semantic features using the associated information to obtain supplementary semantic features;
[0027] The supplementary semantic features are converted into vector representations and fused with the text word vectors of the preprocessed text sequence to obtain the enhanced text vector sequence.
[0028] Preferably, the enhanced text vector sequence is subjected to contextual expansion processing using an attention mechanism to obtain an expanded vector sequence, including:
[0029] The relevance of each word in the preprocessed text sequence to the sentiment classification task is calculated using an attention mechanism to obtain the attention weight;
[0030] By filtering the word vectors corresponding to the top K attention weights, we obtain the key sentiment words;
[0031] Text fragments in the external text library containing the text to be analyzed that have a semantic similarity exceeding the similarity threshold with the key sentiment words are filtered to obtain a retrieval text vector sequence;
[0032] The retrieved text vector sequence and the enhanced text vector sequence are fused to obtain the extended vector sequence.
[0033] Preferably, when multimodal information exists, the extended vector sequence and the multimodal information are fused to obtain a multidimensional fused feature sequence, including:
[0034] Feature extraction is performed on the multimodal information to obtain an emotion feature vector; the multimodal information includes: images and speech;
[0035] Perform dimensional alignment on the sentiment feature vector and the extended vector sequence;
[0036] The emotional feature vector, the extended vector sequence, and the enhanced text vector sequence are weighted and fused to obtain the multi-dimensional fused feature sequence.
[0037] Preferably, the enhanced text feature sequence is input into an improved BERT network for encoding processing to obtain deep semantic features, including:
[0038] By adaptively adjusting the receptive field, each sub-block in the long text sequence of the enhanced text feature sequence is independently encoded and cross-attention fused, while the short text sequence of the enhanced text feature sequence is fully encoded to obtain adaptive receptive coding features; the expression for the adaptive receptive field adjustment is: ;in, For the size of the sensing field; For adaptive parameters; The base length parameter; This represents the actual effective length of the text.
[0039] The adaptive receptive coding features are sequentially subjected to rotational position coding, relative position coding, weighted fusion coding, and position information injection to obtain position-enhanced features;
[0040] The location enhancement features are extracted using a multi-scale convolutional feature fusion network, which performs multi-scale convolutional feature extraction, pooling, feature concatenation, weighted fusion, and residual connection to obtain multi-scale residual fusion features.
[0041] The deep semantic features are obtained by sequentially performing alternating attention calculation, sparse attention filtering, and attention output fusion on the multi-scale residual fusion features; the order of alternating attention calculation is: global attention calculation, local attention calculation, and local attention calculation.
[0042] Preferably, adversarial examples are generated based on the deep semantic features and the real text sentiment tags, and the improved BERT network is optimized through adversarial training using the adversarial examples to obtain the text sentiment classification model, including:
[0043] The deep semantic features are mapped and classified to obtain the network prediction results;
[0044] The original loss is obtained by calculating the preset loss function based on the network prediction results and the real text sentiment tags;
[0045] The improved BERT network is subjected to gradient calculation based on the original loss to obtain the feature gradient;
[0046] Generate targeted adversarial perturbations based on the feature gradient;
[0047] The targeted adversarial perturbation is fused with the embedding layer features of the improved BERT network to obtain the adversarial example embedding representation;
[0048] The adversarial example embedding representation is input into the improved BERT network for computation and loss calculation to obtain the adversarial loss;
[0049] The original loss and the adversarial loss are weighted and fused to obtain the total loss;
[0050] The improved BERT network is iteratively optimized using the total loss to obtain the text sentiment classification model.
[0051] Preferably, a text sentiment classification system based on adversarial training and an improved BERT includes:
[0052] The preprocessing module is used to perform text component recognition, format standardization, semantic resolution and sentiment annotation, and contextual semantic association on the text to be analyzed, so as to obtain a preprocessed text sequence.
[0053] The enhancement processing module is used to perform entity and relation matching, semantic feature supplementation, and vector transformation on the preprocessed text sequence using a pre-built knowledge graph to obtain an enhanced text vector sequence.
[0054] The context extension module is used to perform context extension processing on the enhanced text vector sequence using an attention mechanism to obtain an extended vector sequence.
[0055] A multi-dimensional fusion module is used to fuse the extended vector sequence and the multi-modal information when multi-modal information exists to obtain a multi-dimensional fused feature sequence.
[0056] The length adaptive processing module is used to perform adaptive sequence length processing on the multi-dimensional fused feature sequence according to a length threshold to obtain an enhanced text feature sequence;
[0057] The sentiment classification model is used to perform adaptive length encoding, positional encoding enhancement, multi-scale feature fusion, and attention fusion on the enhanced text feature sequence to obtain the text sentiment classification result.
[0058] Preferably, an electronic device includes: at least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the aforementioned text sentiment classification method based on adversarial training and improved BERT.
[0059] Preferably, a non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the aforementioned text sentiment classification method based on adversarial training and improved BERT.
[0060] The present invention discloses the following technical effects:
[0061] This invention provides a text sentiment classification method, system, electronic device, and storage medium based on adversarial training and an improved BERT. By adaptively adjusting the receptive field and processing the sequence length, it addresses the problems of lost sentiment information in long texts and insufficient semantic representation in short texts found in existing methods, achieving the supplementation of semantic features in short texts and dynamic adjustment of long sequences. Furthermore, through dependency parsing, multi-scale feature fusion, and adaptive receptive encoding features, it overcomes the difficulty in modeling sentiment changes in existing methods, capturing the influence of complex semantics such as negation and transition on sentiment tendency. Attached Figure Description
[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0063] Figure 1 A schematic diagram of the text sentiment classification process based on adversarial training and improved BERT provided for embodiments of the present invention;
[0064] Figure 2 This is a schematic diagram of the text preprocessing process provided in an embodiment of the present invention;
[0065] Figure 3 This is a schematic diagram of the text enhancement process provided in an embodiment of the present invention;
[0066] Figure 4A schematic diagram of the context expansion process provided in an embodiment of the present invention;
[0067] Figure 5 This is a schematic diagram of the multi-dimensional fusion process provided in an embodiment of the present invention. Detailed Implementation
[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0069] The purpose of this invention is to provide a text sentiment classification method, system, electronic device, and storage medium based on adversarial training and an improved BERT, which solves the problems of loss of sentiment information in long texts, insufficient semantic representation of short texts, difficulty in modeling sentiment changes, and difficulty in transfer learning across text lengths in existing methods.
[0070] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0071] Figure 1 This is a schematic diagram of the text sentiment classification process based on adversarial training and improved BERT provided in an embodiment of the present invention, as shown below. Figure 1 As shown, this invention provides a text sentiment classification method based on adversarial training and an improved BERT, comprising:
[0072] Step 100: Perform text component identification, format standardization, semantic resolution and sentiment annotation, and contextual semantic association on the text to be analyzed to obtain a preprocessed text sequence;
[0073] Step 200: Use a pre-constructed knowledge graph to perform entity and relation matching, semantic feature supplementation, and vector transformation on the preprocessed text sequence to obtain an enhanced text vector sequence;
[0074] Step 300: Apply an attention mechanism to the enhanced text vector sequence to perform contextual expansion processing, resulting in an expanded vector sequence;
[0075] Step 400: When multimodal information exists, the extended vector sequence and the multimodal information are fused to obtain a multidimensional fused feature sequence;
[0076] Step 500: Perform adaptive sequence length processing on the multi-dimensional fused feature sequence according to the length threshold to obtain the enhanced text feature sequence;
[0077] Step 600: Using the constructed text sentiment classification model, adaptive length encoding, positional encoding enhancement, multi-scale feature fusion, and attention fusion are applied to the enhanced text feature sequence to obtain the text sentiment classification result; the training process of the text sentiment classification model includes:
[0078] Step 601: Pre-collect target research texts and annotate the target research texts to obtain real text sentiment tags;
[0079] Step 602: Extract the enhanced text feature sequence from the training set;
[0080] Step 603: Input the enhanced text feature sequence into the improved BERT network for encoding processing to obtain deep semantic features;
[0081] Step 604: Generate adversarial examples based on the deep semantic features and the real text sentiment tags, and use the adversarial examples to perform adversarial training and optimization on the improved BERT network to obtain the text sentiment classification model.
[0082] refer to Figure 2 The text to be analyzed undergoes text component identification, format standardization, semantic deconstruction and sentiment annotation, and contextual semantic association processing to obtain a preprocessed text sequence, including:
[0083] Step 101: Use a text component recognition model to perform component recognition on the text to be analyzed, and obtain the component recognition result;
[0084] Step 102: Perform format correction on the component identification results to obtain the corrected component representation;
[0085] Step 103: Annotate the sentiment words in the correction component representation with sentiment based on the pre-constructed sentiment dictionary to obtain sentiment values; the expression for the sentiment value is:
[0086] ;in, The emotional value; For emotional words polarity; Intensity of emotion; For emotional words A set of negative words within a domain; express Elements within;
[0087] Step 104: Use dependency parsing to identify the logical relationships in the text to be analyzed, and perform weighted calculation on the sentiment value of the text to be analyzed based on the logical relationships to obtain the preprocessed text sequence.
[0088] refer to Figure 3 The preprocessed text sequence is subjected to entity and relation matching, semantic feature supplementation, and vector transformation using a pre-constructed knowledge graph to obtain an enhanced text vector sequence, including:
[0089] Step 201: Construct the knowledge graph; the knowledge graph includes: entities, attributes, and relationships;
[0090] Step 202: Match the preprocessed text sequence with the knowledge graph to obtain associated information;
[0091] Step 203: Use the associated information to supplement the semantic features of the preprocessed text sequence to obtain supplementary semantic features;
[0092] Step 204: Convert the supplementary semantic features into vector representations and fuse them with the text word vectors of the preprocessed text sequence to obtain the enhanced text vector sequence.
[0093] refer to Figure 4 The enhanced text vector sequence is subjected to contextual expansion processing using an attention mechanism to obtain an expanded vector sequence, including:
[0094] Step 301: Calculate the relevance of each word in the preprocessed text sequence to the sentiment classification task using an attention mechanism to obtain the attention weight;
[0095] Step 302: Filter the word vectors corresponding to the attention weights of the top K items to obtain key sentiment words;
[0096] Step 303: Filter text segments in the external text library containing the text to be analyzed that have a semantic similarity exceeding the similarity threshold with the key sentiment word to obtain a retrieval text vector sequence;
[0097] Step 304: Merge the retrieved text vector sequence and the enhanced text vector sequence to obtain the extended vector sequence.
[0098] refer to Figure 5 When multimodal information exists, the extended vector sequence and the multimodal information are fused to obtain a multidimensional fused feature sequence, including:
[0099] Step 401: Extract features from the multimodal information to obtain an emotion feature vector; the multimodal information includes: images and speech;
[0100] Step 402: Align the dimensions of the sentiment feature vector and the extended vector sequence;
[0101] Step 403: Perform weighted fusion of the sentiment feature vector, the extended vector sequence, and the enhanced text vector sequence to obtain the multi-dimensional fused feature sequence.
[0102] Specifically, the enhanced text feature sequence is input into an improved BERT network for encoding to obtain deep semantic features, including:
[0103] By adaptively adjusting the receptive field, each sub-block in the long text sequence of the enhanced text feature sequence is independently encoded and cross-attention fused, while the short text sequence of the enhanced text feature sequence is fully encoded to obtain adaptive receptive coding features; the expression for the adaptive receptive field adjustment is: ;in, For the size of the sensing field; For adaptive parameters; The base length parameter; This represents the actual effective length of the text.
[0104] The adaptive receptive coding features are sequentially subjected to rotational position coding, relative position coding, weighted fusion coding, and position information injection to obtain position-enhanced features;
[0105] The location enhancement features are extracted using a multi-scale convolutional feature fusion network, which performs multi-scale convolutional feature extraction, pooling, feature concatenation, weighted fusion, and residual connection to obtain multi-scale residual fusion features.
[0106] The deep semantic features are obtained by sequentially performing alternating attention calculation, sparse attention filtering, and attention output fusion on the multi-scale residual fusion features; the order of alternating attention calculation is: global attention calculation, local attention calculation, and local attention calculation.
[0107] Further, adversarial examples are generated based on the deep semantic features and the real text sentiment tags, and the improved BERT network is optimized through adversarial training using the adversarial examples to obtain the text sentiment classification model, including:
[0108] The deep semantic features are mapped and classified to obtain the network prediction results;
[0109] The original loss is obtained by calculating the preset loss function based on the network prediction results and the real text sentiment tags;
[0110] The improved BERT network is subjected to gradient calculation based on the original loss to obtain the feature gradient;
[0111] Generate targeted adversarial perturbations based on the feature gradient;
[0112] The targeted adversarial perturbation is fused with the embedding layer features of the improved BERT network to obtain the adversarial example embedding representation;
[0113] The adversarial example embedding representation is input into the improved BERT network for computation and loss calculation to obtain the adversarial loss;
[0114] The original loss and the adversarial loss are weighted and fused to obtain the total loss;
[0115] The improved BERT network is iteratively optimized using the total loss to obtain the text sentiment classification model.
[0116] Specifically, the technical solution overview includes: Pre-input enhancement processing: Semantic standardization, knowledge graph enhancement, and context expansion are performed on the original text to output an enhanced text feature sequence; Improved BERT encoding: The enhanced text feature sequence is received and deep semantic features are output through modules such as adaptive length encoding, enhanced position encoding, and multi-scale feature fusion; Adversarial training optimization: Adversarial perturbations are generated based on deep semantic features and fed back to the encoding layer for parameter optimization to improve model robustness; Multi-task classification output: The deep semantic features are used to complete the sentiment classification task and output the final sentiment classification result.
[0117] Furthermore, pre-input text enhancement processing addresses semantic defects and length discrepancies in the original text:
[0118] 1) Using existing text component recognition models, the original text is broken down into components such as date, time, numbers, proper nouns, and sentiment words for recognition; based on the constructed sentiment dictionary (including a dictionary of positive words, a dictionary of negative words, a dictionary of negation words, and a dictionary of degree adverbs), the text is sentiment-annotated, and the sentiment value of each sentiment word is calculated:
[0119]
[0120] in, Words expressing emotions The polarity attribute, where 1 corresponds to positive emotion and -1 corresponds to negative emotion; Used to quantify sentiment words The emotional intensity, ranging from 0 to 1, with a larger value indicating a stronger emotional expression; It is composed of emotional words The set of nearby negative words, Each element in this set represents a specific negative word. Through product operations, the emotional polarity is reversed, ensuring that the polarity of the emotional word is accurately adjusted when a negative word is present.
[0121] Logical relationships in the text are identified through dependency parsing, including: contrast, causality, and progression. Weights are determined based on these logical relationships, and then the relationships are fused and calculated.
[0122]
[0123] in, The total sentiment score, For the number of sentiment words, Let represent the weight of the i-th sentiment word; logical relationships influence weight allocation.
[0124] After sentiment annotation, the preprocessed text sequence is output.
[0125] 2) Semantic enhancement processing based on knowledge graphs:
[0126] Pre-construct a causal deep semantic knowledge graph containing entities, attributes, and relationships;
[0127] The entities in the preprocessed text sequence are matched with the entities in the knowledge graph to extract associated attributes and relationships;
[0128] Based on the association information of knowledge graphs, missing semantic features are supplemented to the text;
[0129] The supplemented semantic features are converted into vector representations and then fused with text word vectors.
[0130]
[0131] in, To enhance text vector sequences, For word vectors, Embed vectors into knowledge graphs. For weight fusion.
[0132] (3) Context extension processing based on attention mechanism:
[0133] The relevance weight of each word to the sentiment classification task is calculated using an attention mechanism. :
[0134]
[0135]
[0136] in, This is a query vector for the sentiment classification task.
[0137] The word vectors with the top K attention weights are retained to highlight key sentiment information;
[0138] Based on key sentiment words, semantically similar text fragments are retrieved from an external text database. Contextual information is then expanded, resulting in the following expanded vector sequence:
[0139]
[0140] in, For an extended vector sequence, To enhance text vector sequences, A vector sequence for retrieving text; For connection operations.
[0141] 4) Multi-dimensional feature fusion processing:
[0142] When the text contains multimodal information such as images and speech, the corresponding sentiment feature vectors are extracted and converted into a representation with the same dimension as the text vector;
[0143] Align the dimensions of text vectors, knowledge graph vectors, and multimodal feature vectors to ensure consistency in fusion;
[0144] Multi-dimensional features are fused into a unified feature vector using learnable weights;
[0145] The co-occurrence frequency of sentiment words with degree adverbs and negation words is statistically analyzed, and a co-occurrence feature vector is constructed. This vector is then concatenated with the fusion vector to obtain a multi-dimensional fusion feature sequence.
[0146] 5) Adaptive text length processing:
[0147] Set token length threshold If the length of the token corresponding to the multi-dimensional fused feature sequence is... If it is determined to be a long text, then it is a short text.
[0148] Intelligent block segmentation is performed based on sentence boundaries and semantic integrity to ensure that the length of each sub-block does not exceed a preset token threshold, and semantic association information between blocks is preserved after segmentation.
[0149] Pooling is performed on the fused feature sequence of short texts to condense key sentiment features while retaining expanded contextual information, ensuring that the feature dimensions are consistent with those of long text sub-blocks.
[0150] The feature sequences of each sub-block of long text and the feature sequences of short text are used to generate an enhanced text feature sequence in a unified format.
[0151] Specifically, the improved BERT architecture design:
[0152] 1) Adaptive length encoding mechanism: Each sub-block of long text is encoded independently, while semantic relationships between sub-blocks are captured through a cross-block attention mechanism; a single sequence of short text is fully encoded, and semantics are supplemented using expanded contextual information; and a learnable length adaptive parameter is used. Adjusting the size of the receptive field of the coding layer When encoding long texts, the local attention window is enlarged; when encoding short texts, the global attention weights are increased.
[0153]
[0154] in, The actual effective length of the text. For the pre-defined base length parameter, These are adaptive parameters.
[0155] 2) Enhanced positional encoding module: Replaces traditional absolute positional encoding with Rotated Positional Encoding (RoPE), encodes positional information through multiplication, and supports long context expansion; calculates the relative distance between words to generate relative positional encoding vectors; weightedly fuses RoPE encoding and relative positional encoding to enhance positional information representation; and fuses enhanced positional encoding with feature sequences to generate position-enhanced features with positional information.
[0156] 3) Multi-scale feature fusion network: Use 1D convolutional kernels of different sizes to extract local sentiment features at different scales; perform max pooling and average pooling on each convolutional output to extract key features; concatenate pooling features of different scales and fuse them into unified features through learnable weights; perform residual connection between the fused features and the original encoded features to obtain multi-scale residual fusion features.
[0157] 4) Improved attention mechanisms:
[0158] Alternating Attention Pattern: Employs a global-local-local alternating attention structure. In every three attention layers, the first layer uses global attention to capture long-range dependencies, while the latter two layers use local window attention to improve efficiency.
[0159] Global attention :
[0160]
[0161] in, For the Softmax function; , , These are the query matrix, key matrix, and value matrix, respectively. For dimensions.
[0162] Local attention Each location only considers the context within w surrounding windows:
[0163]
[0164] in, Let i be the query vector corresponding to position i. Let i be the key vector from position i−w to position i+w. Let i be the value vector from position i−w to position i+w.
[0165] Low-weight attention connections are filtered out by a dynamic threshold, and only connections with weights higher than the dynamic threshold are retained, reducing computational complexity; the outputs of global attention and local attention are weighted and fused to output deep semantic features.
[0166] Preferably, the adversarial training mechanism generates adversarial examples by performing sentiment classification prediction based on deep semantic features and calculating the original loss. :
[0167]
[0168]
[0169] in, For network prediction results, and For classifier weights and biases, For the number of emotion categories, Cross-entropy loss; This is the actual label.
[0170] Calculate the gradient of the loss function with respect to the embedding layer features, generate targeted adversarial perturbations based on the gradients, and control the perturbation amplitude to avoid excessive distortion; add the perturbations to the embedding layer features to generate adversarial example embedding representations.
[0171] Adversarial training optimization: Adversarial examples are embedded into the improved BERT architecture. The above encoding process outputs deep semantic features of adversarial examples. Sentiment classification prediction is performed based on these features, and adversarial loss is calculated. Total loss calculation: The original loss and adversarial loss are weighted to avoid overfitting or underfitting of the model. Backpropagation is performed based on the total loss to update the improved BERT architecture, classifier, and learnable parameters in the input preprocessing, and the optimized model parameters are output.
[0172] As an optional implementation, this embodiment also provides a text sentiment classification system based on adversarial training and an improved BERT, including:
[0173] The preprocessing module is used to perform text component recognition, format standardization, semantic resolution and sentiment annotation, and contextual semantic association on the text to be analyzed, so as to obtain a preprocessed text sequence.
[0174] The enhancement processing module is used to perform entity and relation matching, semantic feature supplementation, and vector transformation on the preprocessed text sequence using a pre-built knowledge graph to obtain an enhanced text vector sequence.
[0175] The context extension module is used to perform context extension processing on the enhanced text vector sequence using an attention mechanism to obtain an extended vector sequence.
[0176] A multi-dimensional fusion module is used to fuse the extended vector sequence and the multi-modal information when multi-modal information exists to obtain a multi-dimensional fused feature sequence.
[0177] The length adaptive processing module is used to perform adaptive sequence length processing on the multi-dimensional fused feature sequence according to a length threshold to obtain an enhanced text feature sequence;
[0178] The sentiment classification model is used to perform adaptive length encoding, positional encoding enhancement, multi-scale feature fusion, and attention fusion on the enhanced text feature sequence to obtain the text sentiment classification result.
[0179] As an optional implementation, this embodiment also provides an electronic device, including: at least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the aforementioned text sentiment classification method based on adversarial training and improved BERT.
[0180] As an optional implementation, this embodiment also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the aforementioned text sentiment classification method based on adversarial training and improved BERT.
[0181] The beneficial effects of this invention are as follows:
[0182] This invention reduces the loss of information in long texts and supplements the semantic features of short texts by adaptive adjustment of receptive field, adaptive sequence length processing, and adaptive adjustment of receptive field, thereby improving the integrity of emotional information and the accuracy of semantic representation in both types of texts. Through dependency parsing, multi-scale feature fusion, and adaptive receptive coding features, it can capture the influence of complex semantics such as negation and transition on sentiment, effectively model dynamic sentiment changes, and make sentiment judgment more consistent with the true intent of the text.
[0183] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0184] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A text sentiment classification method based on adversarial training and an improved BERT, characterized in that, include: The text to be analyzed is subjected to text component identification, format standardization, semantic decomposition and sentiment annotation, and contextual semantic association to obtain a preprocessed text sequence. The pre-constructed knowledge graph is used to perform entity and relation matching, semantic feature supplementation, and vector transformation on the pre-processed text sequence to obtain an enhanced text vector sequence. The enhanced text vector sequence is subjected to context expansion processing using an attention mechanism to obtain an expanded vector sequence. When multimodal information exists, the extended vector sequence and the multimodal information are fused to obtain a multidimensional fused feature sequence; The multi-dimensional fused feature sequence is subjected to adaptive sequence length processing based on a length threshold to obtain an enhanced text feature sequence; The enhanced text feature sequence is subjected to adaptive length encoding, positional encoding enhancement, multi-scale feature fusion, and attention fusion using the constructed text sentiment classification model to obtain the text sentiment classification result. The training process of the text sentiment classification model includes: Pre-collect target research texts and annotate the target research texts to obtain real text sentiment tags; Extract the enhanced text feature sequence from the training set; The enhanced text feature sequence is input into the improved BERT network for encoding to obtain deep semantic features; Adversarial examples are generated based on the deep semantic features and the real text sentiment tags, and the improved BERT network is trained and optimized using the adversarial examples to obtain the text sentiment classification model.
2. The text sentiment classification method based on adversarial training and improved BERT according to claim 1, characterized in that, The text to be analyzed undergoes text component identification, format standardization, semantic deconstruction and sentiment annotation, and contextual semantic association processing to obtain a preprocessed text sequence, including: The text to be analyzed is analyzed using a text component recognition model to identify the components and obtain the component recognition results. The format of the component identification results is corrected to obtain the corrected component representation; Sentiment tags are applied to the sentiment words in the correction component representation based on a pre-constructed sentiment dictionary to obtain sentiment values; the expression for the sentiment values is: ;in, The emotional value; For emotional words polarity; Intensity of emotion; For emotional words A set of negative words within a domain; express Elements within; The logical relationships of the text to be analyzed are identified using dependency parsing, and the sentiment values of the text to be analyzed are weighted and calculated based on the logical relationships to obtain the preprocessed text sequence.
3. The text sentiment classification method based on adversarial training and improved BERT according to claim 1, characterized in that, The preprocessed text sequence is subjected to entity and relation matching, semantic feature supplementation, and vector transformation using a pre-constructed knowledge graph to obtain an enhanced text vector sequence, including: Construct the knowledge graph; the knowledge graph includes: entities, attributes, and relationships; The preprocessed text sequence is matched with the knowledge graph to obtain associated information; The preprocessed text sequence is supplemented with semantic features using the associated information to obtain supplementary semantic features; The supplementary semantic features are converted into vector representations and fused with the text word vectors of the preprocessed text sequence to obtain the enhanced text vector sequence.
4. The text sentiment classification method based on adversarial training and improved BERT according to claim 1, characterized in that, The enhanced text vector sequence is subjected to context expansion processing using an attention mechanism to obtain an expanded vector sequence, including: The relevance of each word in the preprocessed text sequence to the sentiment classification task is calculated using an attention mechanism to obtain the attention weight; By filtering the word vectors corresponding to the top K attention weights, we obtain the key sentiment words; Text fragments in the external text library containing the text to be analyzed that have a semantic similarity exceeding the similarity threshold with the key sentiment words are filtered to obtain a retrieval text vector sequence; The retrieved text vector sequence and the enhanced text vector sequence are fused to obtain the extended vector sequence.
5. The text sentiment classification method based on adversarial training and improved BERT according to claim 1, characterized in that, When multimodal information exists, the extended vector sequence and the multimodal information are fused to obtain a multidimensional fused feature sequence, including: Feature extraction is performed on the multimodal information to obtain an emotion feature vector; the multimodal information includes: images and speech; Perform dimensional alignment on the sentiment feature vector and the extended vector sequence; The emotional feature vector, the extended vector sequence, and the enhanced text vector sequence are weighted and fused to obtain the multi-dimensional fused feature sequence.
6. The text sentiment classification method based on adversarial training and improved BERT according to claim 1, characterized in that, The enhanced text feature sequence is input into the improved BERT network for encoding to obtain deep semantic features, including: By adaptively adjusting the receptive field, each sub-block in the long text sequence of the enhanced text feature sequence is independently encoded and cross-attention fused, while the short text sequence of the enhanced text feature sequence is fully encoded to obtain adaptive receptive coding features; the expression for the adaptive receptive field adjustment is: ;in, For the size of the sensing field; For adaptive parameters; The base length parameter; This represents the actual effective length of the text. The adaptive receptive coding features are sequentially subjected to rotational position coding, relative position coding, weighted fusion coding, and position information injection to obtain position-enhanced features; The location enhancement features are extracted using a multi-scale convolutional feature fusion network, which performs multi-scale convolutional feature extraction, pooling, feature concatenation, weighted fusion, and residual connection to obtain multi-scale residual fusion features. The deep semantic features are obtained by sequentially performing alternating attention calculation, sparse attention filtering, and attention output fusion on the multi-scale residual fusion features; the order of alternating attention calculation is: global attention calculation, local attention calculation, and local attention calculation.
7. The text sentiment classification method based on adversarial training and improved BERT according to claim 1, characterized in that, Adversarial examples are generated based on the deep semantic features and the real text sentiment tags, and the improved BERT network is trained and optimized using the adversarial examples to obtain the text sentiment classification model, including: The deep semantic features are mapped and classified to obtain the network prediction results; The original loss is obtained by calculating the preset loss function based on the network prediction results and the real text sentiment tags; The improved BERT network is subjected to gradient calculation based on the original loss to obtain the feature gradient; Generate targeted adversarial perturbations based on the feature gradient; The targeted adversarial perturbation is fused with the embedding layer features of the improved BERT network to obtain the adversarial example embedding representation; The adversarial example embedding representation is input into the improved BERT network for computation and loss calculation to obtain the adversarial loss; The original loss and the adversarial loss are weighted and fused to obtain the total loss; The improved BERT network is iteratively optimized using the total loss to obtain the text sentiment classification model.
8. A text sentiment classification system based on adversarial training and an improved BERT, characterized in that, include: The preprocessing module is used to perform text component recognition, format standardization, semantic resolution and sentiment annotation, and contextual semantic association on the text to be analyzed, so as to obtain a preprocessed text sequence. The enhancement processing module is used to perform entity and relation matching, semantic feature supplementation, and vector transformation on the preprocessed text sequence using a pre-built knowledge graph to obtain an enhanced text vector sequence. The context extension module is used to perform context extension processing on the enhanced text vector sequence using an attention mechanism to obtain an extended vector sequence. A multi-dimensional fusion module is used to fuse the extended vector sequence and the multi-modal information when multi-modal information exists to obtain a multi-dimensional fused feature sequence. The length adaptive processing module is used to perform adaptive sequence length processing on the multi-dimensional fused feature sequence according to a length threshold to obtain an enhanced text feature sequence; The sentiment classification model is used to perform adaptive length encoding, positional encoding enhancement, multi-scale feature fusion, and attention fusion on the enhanced text feature sequence to obtain the text sentiment classification result.
9. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform a text sentiment classification method based on adversarial training and improved BERT as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute a text sentiment classification method based on adversarial training and improved BERT as described in any one of claims 1 to 7.