A method and system for automatically labeling sentiment data for fine-grained sentiment analysis

Abstract

The application provides a kind of sentiment data automatic labeling method and system for fine-grained sentiment analysis, belongs to artificial intelligence and sentiment analysis field.The method extracts text, speech and visual features, carries out coarse-grained sentiment pre-classification and confidence weighted fusion;Combined with large language model reasoning and spectral clustering, fine-grained sentiment recognition is realized;The results are optimized by using conflict resolution mechanism, and the final label is generated by multi-level voting integration;Finally, through multi-modal consistency, feature space outlier degree and conflict resolution strategy effect multiple dimensions, the credibility of the final label is evaluated, and low credibility samples are identified.The application realizes the gradual analysis from coarse-grained to fine-grained, effectively solves the problems of modal heterogeneity, information conflict and low labeling credibility, reduces the cost of artificial labeling, and improves the accuracy and reliability of sentiment analysis.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and sentiment analysis, and in particular relates to an automatic labeling method and system for sentiment data for fine-grained sentiment analysis. Background Technology

[0002] With the rapid development of social media, human-computer interaction systems, and smart terminals, multimodal sentiment data, including text, voice, and visual information, is experiencing explosive growth. As an important intersection of artificial intelligence and cognitive science, sentiment computing has significant application value in areas such as public opinion analysis, mental health assessment, intelligent customer service, and personalized recommendations.

[0003] Traditional sentiment analysis methods primarily rely on single-modal data, such as text-based sentiment analysis or speech-based sentiment recognition. However, human emotional expression is complex and multidimensional, and a single modality often cannot fully and accurately reflect the true emotional state. For example, a user may speak an angry word in a calm tone, or express sadness with a smiling expression. Therefore, multimodal sentiment analysis has become a key technological approach to improve the accuracy of sentiment recognition.

[0004] Existing technologies have proposed various solutions for multimodal sentiment analysis of complex emotions. However, current multimodal sentiment analysis methods still face the following technical challenges:

[0005] 1. Different modalities of data (text, speech, vision) have different feature representations, data structures, and semantic granularities. Achieving effective cross-modal feature alignment and semantic fusion is a core challenge. Existing methods typically employ simple feature concatenation or post-fusion strategies, which struggle to capture deep cross-modal semantic relationships.

[0006] 2. In multimodal sentiment analysis, the emotional information conveyed by different modalities may contradict each other. Existing methods lack effective conflict detection and resolution mechanisms and cannot intelligently determine which modality should dominate in a specific context;

[0007] 3. Fine-grained sentiment annotation requires professional annotators to manually annotate multimodal data, which is costly, inefficient, and easily affected by subjective factors. Existing automatic annotation methods mostly rely on large-scale labeled data, making it difficult to adapt to real-world scenarios where annotation resources are scarce.

[0008] 4. Existing automatic labeling methods typically lack quantitative evaluation of the reliability of generated labels, making it difficult to identify low-quality labeled samples and affecting the training effect of downstream models;

[0009] 5. It is difficult to predict fine-grained sentiment categories directly from raw multimodal data, and there is a lack of a progressive reasoning mechanism from coarse-grained sentiment cognition to fine-grained sentiment discrimination.

[0010] Therefore, there is an urgent need for an innovative method that can efficiently handle modal heterogeneity, intelligently resolve modal conflicts, achieve fine-grained automatic sentiment labeling, and evaluate the credibility of labels. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and provide an automatic labeling method and system for sentiment data for fine-grained sentiment analysis. By using deep learning technology to achieve end-to-end closed-loop automatic labeling of multimodal sentiment data through progressive identification and conflict resolution from coarse to fine granularity, the invention enhances the credibility and interpretability of the labeling results and reduces manual costs.

[0012] To achieve the above objectives, the present invention provides the following technical solution:

[0013] This invention provides an automatic sentiment data annotation method for fine-grained sentiment analysis, the method comprising:

[0014] Collect raw multimodal data containing text, speech, and visual data, and use pre-trained deep learning models to extract high-dimensional semantic features of each modality;

[0015] Based on the high-dimensional semantic features of each modality, a three-class classification process of positive, negative and neutral sentiment is performed to obtain the pre-classified coarse-grained sentiment labels and corresponding classification confidence scores for each modality.

[0016] The fusion weight of each modality is dynamically calculated based on the classification confidence of each modality. The high-dimensional semantic features of each modality are then weighted and fused using the fusion weight to generate a fusion feature matrix.

[0017] Based on the fusion feature matrix and pre-classified coarse-grained sentiment tags, combined with pre-trained large language model inference and spectral clustering methods, fine-grained sentiment tags containing 11 categories are predicted.

[0018] The system detects and resolves conflicts between the coarse-grained labels pre-classified by each modality and the coarse-grained labels predicted by the large model, and iteratively optimizes the coarse-grained sentiment classification results through loss function.

[0019] Fine-grained sentiment tags from different prediction sources are integrated, and a weighted voting mechanism is used to generate the final fine-grained sentiment tags, as well as the coarse-grained sentiment tags corresponding to the fine-grained sentiment tags.

[0020] A comprehensive evaluation system is constructed from multiple dimensions, including multimodal consistency, outlier degree in feature space, and decision-making effectiveness in conflict resolution, to quantify the credibility of the final label and identify low-credibility samples.

[0021] In one implementation, the step of extracting high-dimensional semantic features of each modality using a pre-trained deep learning model includes:

[0022] Using a pre-trained text parameter feature model, the input text sequence Encoded as fixed-dimensional text features Among them, individual text features BERT is a pre-trained text parameter feature extraction model function. For text sequences The first in Each text unit , The length of the sampling sequence. For text feature dimensions, For dimension The set of real matrices, For length is The set of real column vectors;

[0023] Using a pre-trained speech parameter feature model, from the input audio signal Extracting speech features Among them, individual speech features , Pre-trained speech feature extraction model function, audio signal The first in One audio signal unit, , For speech feature dimension, For dimension The set of real matrices, For length is The set of real column vectors;

[0024] Using a pre-trained visual feature model, corresponding to the input image or video frame sequence Extracting spatial and temporal visual features Among them, individual image visual features , For pre-trained image visual feature extraction model functions, single video visual features , For pre-trained video visual feature extraction model functions, For visual feature dimensions, For dimension The set of real matrices, For length is The set of real column vectors, A sequence of image or video frames The first in Each image or video unit .

[0025] In one implementation, the pre-classified coarse-grained sentiment tag is represented as:

[0026] ,

[0027] in, For each modality, pre-classify coarse-grained sentiment labels, with subscripts... These correspond to three different modalities: text features, speech features, and visual features. Pos represents positive sentiment, Neg represents negative sentiment, and Neu represents neutral sentiment. To use the sequence length;

[0028] The classification confidence level is expressed as follows: ,

[0029] in, For the confidence vector, the subscripts are... These correspond to three different modalities: text features, speech features, and visual features.

[0030] In one implementation, the step of dynamically calculating the fusion weight of each modality based on the classification confidence of each modality, and using the fusion weight to perform weighted fusion processing on the high-dimensional semantic features of each modality to generate a fusion feature matrix includes:

[0031] For the original confidence vector , , Perform Softmax normalization, subscript These correspond to three different modalities: text features, speech features, and visual features.

[0032] Temperature parameters are introduced to adjust the smoothness of the weight distribution, and the fusion weights for each mode are set. for:

[0033] ,

[0034] in, It is a natural exponential function. For temperature parameters, For the first The confidence level probability value of each modality A marker for traversing all modalities; For the mode whose weights are currently being calculated, For each traversed modality;

[0035] Based on the calculated fusion weights , , Text features respectively Speech features Visual features Weighted fusion is performed to obtain the fusion feature matrix. The fused feature matrix .

[0036] In one implementation, the method of predicting fine-grained sentiment tags comprising 11 categories based on a fused feature matrix and pre-classified coarse-grained sentiment tags, combined with pre-trained large language model inference and spectral clustering, includes:

[0037] The fused feature matrix is ​​converted into a natural language description vector through a lightweight semantic mapping network, while the pre-classified coarse-grained sentiment labels of each modality are mapped to natural language descriptions.

[0038] Based on the preset prompt word template, natural language description vectors and coarse-grained sentiment label descriptions are integrated to construct prompt words that are input into the pre-trained large language model;

[0039] Input the prompt words into a pre-trained large language model to obtain fine-grained sentiment labels from the model's predicted output;

[0040] Establish The mapping function maps the fine-grained sentiment labels predicted by the large model to the coarse-grained sentiment labels predicted by the large model; where, It refers to the mapping from an 11-dimensional real number space to a 3-dimensional real number space;

[0041] Spectral clustering was performed on the fused feature matrix to obtain fine-grained sentiment tags containing 11 categories.

[0042] In one implementation, the mapping function is defined as follows:

[0043] ,

[0044] in, For mapping functions, The input variables for fine-grained sentiment labels are: Happy (pleasure), Angry (anger), Sadness (sadness), Fear (fear), Surprise (surprise), Disgust (disgust), Frustration (frustration), Confused (confusion), Anxious (anxiety), Startled (startle), and Neutral (neutral).

[0045] In one implementation, the step of detecting and resolving conflicts between the coarse-grained labels pre-classified for each modality and the coarse-grained labels predicted by the large model, and iteratively optimizing the coarse-grained sentiment classification results through a loss function, includes:

[0046] For any sampling time Coarse-grained sentiment tags for each modality pre-classification , , And coarse-grained sentiment labels predicted by large models If they are completely consistent, they are determined to be without conflict, and the consistent label is directly used as the coarse classification label after conflict resolution; if they are inconsistent, they are determined to be in conflict.

[0047] in, , , The three single-modal models, namely text, speech, and vision, are respectively in the [number]th [year]. Real-time output of coarse-grained sentiment pre-classification tags;

[0048] For cases deemed conflicting, a weighted voting decision is made using the fusion weights corresponding to each modality to obtain the corresponding sampling time. Conflict resolution coarse classification tags Its expression is:

[0049] ,

[0050] in, This represents the weighted sum of the most sentiment categories. For the first At this moment, the first The fusion weights of each modality; This is an indicator function used to identify whether the label category matches; For the first At this moment, the pre-classification coarse-grained sentiment tags for each modality, For a set of modal labels, ; A collection of emotional tags ;

[0051] The cross-entropy loss function is used to update the model parameters of the coarse classifier and the parameters of the pre-trained deep learning model, thereby achieving iterative optimization of the coarse-grained sentiment classification results. The mathematical expression is:

[0052] ,

[0053] in, For the first At this moment, the first Coarse-grained sentiment tags with manual annotations corresponding to each modality. This represents taking the natural logarithm of the confidence probability value.

[0054] In one implementation, the final fine-grained sentiment tags are generated through a weighted voting mechanism. for:

[0055] ;

[0056] in, , The voting weights are hyperparameters. For the first Fine-grained sentiment tags for real-time clustering. For the first Fine-grained sentiment tags in the time-major model prediction output;

[0057] Using a mapping function, the final fine-grained sentiment labels are mapped to the final coarse-grained labels.

[0058] In one implementation, for any time The calculation process and credibility determination process for the multi-dimensional evaluation indicators of the sample are as follows:

[0059] Calculate the multimodal consistency score:

[0060] ;

[0061] in, For the first Multimodal consistency score of samples at time step;

[0062] Calculate the outlier score in the feature space:

[0063] ;

[0064] in, For the first Outlier score in the feature space of the sample at time step. It is a natural exponential function. The Mahalanobis distance, , , For the fusion feature matrix The corresponding mean and variance; The inverse of the covariance matrix, This represents the transpose of the sample vector and the deviation vector from the center.

[0065] Calculate the conflict resolution decision score:

[0066] ;

[0067] in, For the first Conflict resolution decision score for time-series samples This means that when there is no conflict, the weighted voting score for each sentiment category k is calculated, and the maximum value is taken as the decision score at that moment. Indicates when the first Execute this branch if there are no conflicts at any given time.

[0068] After obtaining the scores in these three dimensions, hyperparameter weights are introduced to calculate the final overall credibility of the sample:

[0069] ;

[0070] in, Indicates the first The final overall credibility of the time-based samples. For hyperparameter weights, This represents the scores across three dimensions. ;

[0071] Set a confidence threshold. When the overall confidence of a sample is lower than the threshold, the sample is determined to be a low-confidence sample.

[0072] This invention also provides an automatic sentiment data annotation system for fine-grained sentiment analysis, used to perform the above-described method, the system comprising:

[0073] The multimodal feature extraction module is used to collect raw multimodal data containing text, speech and visual data, and to extract high-dimensional semantic features of each modality using a pre-trained deep learning model;

[0074] The coarse-grained sentiment classification module is used to perform three-class classification processing (positive, negative, and neutral) based on the high-dimensional semantic features of each modality, and obtain the pre-classified coarse-grained sentiment labels and corresponding classification confidence scores for each modality.

[0075] The feature fusion module is used to dynamically calculate the fusion weight of each modality based on the classification confidence of each modality, and use the fusion weight to perform weighted fusion processing on the high-dimensional semantic features of each modality to generate a fusion feature matrix;

[0076] The fine-grained sentiment recognition module is used to predict fine-grained sentiment tags containing 11 categories based on the fused feature matrix and pre-classified coarse-grained sentiment tags, combined with pre-trained large language model reasoning and spectral clustering methods.

[0077] The conflict resolution and iterative update module is used to detect and resolve conflicts between the coarse-grained labels preclassified by each modality and the coarse-grained labels predicted by the large model, and to iteratively optimize the coarse-grained sentiment classification results through the loss function.

[0078] The multi-level voting integration module is used to integrate fine-grained sentiment tags from different prediction sources, and generate the final fine-grained sentiment tags and corresponding coarse-grained sentiment tags through a weighted voting mechanism.

[0079] The confidence assessment module is used to construct a comprehensive assessment system from multiple dimensions, including multimodal consistency, feature space outlier degree, and conflict resolution decision effect, to quantify the confidence of the final label and identify low-confidence samples.

[0080] The beneficial effects of this invention are as follows: This invention achieves end-to-end automatic annotation of multimodal sentiment data through a progressive process combining multimodal feature extraction, coarse-grained pre-classification, dynamic weighted fusion, and fine-grained recognition, significantly reducing manual costs; it introduces a confidence-based conflict resolution mechanism and a multimodal semantic alignment strategy to effectively solve the problems of modal heterogeneity and sentiment expression conflict; it combines large language model reasoning and spectral clustering analysis to improve the accuracy and interpretability of fine-grained sentiment discrimination, and outputs quantitative confidence indicators based on multi-dimensional credibility assessment, enhancing the reliability of the annotation results. It can be widely applied to scenarios with high requirements for sentiment computing, such as public opinion analysis and intelligent customer service. Attached Figure Description

[0081] The accompanying drawings, as part of this invention, are provided to further illustrate the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation thereof. Clearly, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0082] Figure 1 This is a flowchart of an automatic sentiment data annotation method for fine-grained sentiment analysis provided in an embodiment of the present invention;

[0083] Figure 2 This is an architecture diagram of an automatic sentiment data annotation system for fine-grained sentiment analysis provided in an embodiment of the present invention;

[0084] Figure 3 This is an architecture diagram of the fusion feature matrix calculation module provided in one embodiment of the present invention;

[0085] Figure 4 This is an architecture diagram of conflict resolution and closed-loop iterative update provided in one embodiment of the present invention.

[0086] It should be noted that these accompanying drawings and textual descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art by referring to specific embodiments. Detailed Implementation

[0087] To make the objectives and technical solutions of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0089] In online psychological counseling and health monitoring platforms, clients communicate with AI psychological assistants or human counselors via video conversations. The communication process generates multimodal data including text (chat logs or voice-to-text), speech (tone, speed, volume), and visual (facial expressions, body language). Accurately identifying the client's fine-grained emotional state (e.g., 11 categories such as anxiety, frustration, confusion, surprise, etc.) is crucial for assessing their psychological state, providing personalized intervention suggestions, and issuing crisis warnings. However, current platforms mostly rely on counselors' subjective judgment or single-text sentiment analysis, resulting in low efficiency, poor consistency, and an inability to automate the large-scale processing of historical conversation data for model training and effectiveness evaluation. Therefore, this invention proposes an automatic sentiment data annotation method for fine-grained sentiment analysis.

[0090] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0091] Reference Figure 1 As shown in one embodiment, an automatic sentiment data annotation method for fine-grained sentiment analysis is provided, which includes the following steps:

[0092] Step S100: Collect raw multimodal data containing text, speech and visual data, and use a pre-trained deep learning model to extract high-dimensional semantic features of each modality.

[0093] High-dimensional semantic features refer to numerical vectors (or matrices) extracted from raw multimodal data such as text, speech, and vision, which contain rich semantic information and have high dimensionality. Their core function is to transform unstructured raw data into structured data that can be processed by computers, while retaining key semantic information of emotional expression.

[0094] In this embodiment of the application, high-dimensional semantic features of each modality are extracted using a pre-trained deep learning model, including:

[0095] Using a pre-trained text parameter feature model, the input text sequence Encoded as fixed-dimensional text features Among them, individual text features BERT is a pre-trained text parameter feature extraction model function. For text sequences The first in Each text unit , The length of the sampling sequence. For text feature dimensions, For dimension The set of real matrices, For length is The set of real column vectors;

[0096] Using a pre-trained speech parameter feature model, from the input audio signal Extracting speech features Among them, individual speech features , Pre-trained speech feature extraction model function, audio signal The first in One audio signal unit, , For speech feature dimension, For dimension The set of real matrices, For length is The set of real column vectors;

[0097] Using a pre-trained visual feature model, corresponding to the input image or video frame sequence Extracting spatial and temporal visual features Among them, individual image visual features , For pre-trained image visual feature extraction model functions, single video visual features , For pre-trained video visual feature extraction model functions, For visual feature dimensions, For dimension The set of real matrices, For length is The set of real column vectors, A sequence of image or video frames The first in Each image or video unit .

[0098] The implementation logic of the pre-trained model is as follows: load the weights and network structure that have been trained by the official team → perform standardization preprocessing on the input data → extract the feature vectors output by the model.

[0099] Specifically, the pre-trained text parameter feature model is a pre-trained BERT model, the pre-trained speech parameter feature model is a pre-trained wav2vec 2.0 model, and the pre-trained visual feature model is a pre-trained Vision Transformer (ViT) model or a pre-trained TimeSformer model. Target social media content is crawled, containing text, speech, and visual data (such as facial expression images and body movement video frames). Text features are extracted using the pre-trained models. Speech features Visual features .

[0100] Step S200: Based on the high-dimensional semantic features of each modality, perform three-class classification processing of sentiment tendency: positive, negative, and neutral, to obtain the pre-classified coarse-grained sentiment labels and corresponding classification confidence scores for each modality.

[0101] High-dimensional semantic features based on each modality , , It performs a quick and preliminary trinomial classification of sentiment into positive (Pos), negative (Neg), and neutral (Neu), and outputs the classification confidence score.

[0102] The text features can be processed by three independent classifiers: a text coarse classifier, a speech coarse classifier, and a visual coarse classifier. Speech features Visual features Perform three-category classification.

[0103] Furthermore, the pre-classified coarse-grained sentiment labels are represented as follows:

[0104] ,

[0105] in, For each modality, pre-classify coarse-grained sentiment labels, with subscripts... These correspond to three different modalities: text features, speech features, and visual features. Pos represents positive sentiment, Neg represents negative sentiment, and Neu represents neutral sentiment. To use the sequence length;

[0106] Classification confidence is expressed as: ,

[0107] in, For the confidence vector, the subscripts are... These correspond to three different modalities: text features, speech features, and visual features.

[0108] Step S300: Dynamically calculate the fusion weight of each modality based on the classification confidence of each modality, and use the fusion weight to perform weighted fusion processing on the high-dimensional semantic features of each modality to generate a fusion feature matrix.

[0109] In this embodiment of the application, step S300 specifically includes the following steps:

[0110] Step S310: For the original confidence vector , , Perform Softmax normalization, subscript These correspond to three different modalities: text features, speech features, and visual features.

[0111] Step S320: Introduce temperature parameters to adjust the smoothness of weight distribution and set the fusion weights for each mode. for:

[0112] ,

[0113] in, It is a natural exponential function. For temperature parameters, For the first The confidence level probability value of each modality A marker for traversing all modalities; For the mode whose weights are currently being calculated, For each traversed modality.

[0114] Step S330: Based on the calculated fusion weights , , Text features respectively Speech features Visual features Weighted fusion is performed to obtain the fusion feature matrix. The fused feature matrix .

[0115] Specifically, the system receives the consultation conversation stream in real time, and performs data input and feature extraction. Three independent classifiers are used to process the text features. Speech features Visual features Perform a quick analysis and output a "positive / negative / neutral" prediction result. and confidence level Based on these confidence levels, the fusion weights for each modality are calculated using Softmax normalization. The original features are then weighted and summed to generate a fusion feature matrix that reflects the reliability of each modality. .

[0116] Step S400: Based on the fused feature matrix and pre-classified coarse-grained sentiment tags, combined with pre-trained large language model inference and spectral clustering methods, predict fine-grained sentiment tags containing 11 categories.

[0117] Step S410: Fuse the feature matrix A lightweight semantic mapping network is used to convert the data into natural language description vectors, and pre-classify coarse-grained labels for each modality. , , Mapped to natural language description.

[0118] Step S420: Based on the preset prompt word template, integrate the above natural language description vector and coarse-grained sentiment label description to construct prompt words that are input to the pre-trained large language model.

[0119] Alternatively, a prompt template can be manually set based on the scenario and task, or the large language model can generate a range of prompts for the scenario first. For example, prompts from the large language model could be like this: "The scene is crowded and noisy, the background music is rousing, the text content is a call for protest; please determine the dominant emotion."

[0120] Step S430: Input the prompt words into the pre-trained large language model to obtain the coarse-grained and fine-grained sentiment tags predicted by the large model. .

[0121] Step S440: Establish The mapping function maps the fine-grained sentiment labels predicted by the large model to the coarse-grained sentiment labels predicted by the large model. .in, It refers to the mapping from an 11-dimensional real number space to a 3-dimensional real number space.

[0122] Step S450: For the fused feature matrix Perform spectral clustering to obtain fine-grained sentiment tags with clusters of 11 categories. .

[0123] Furthermore, the mapping function is defined as follows:

[0124] ,

[0125] in, For mapping functions, The input variables for fine-grained sentiment labels are: Happy (pleasure), Angry (anger), Sadness (sadness), Fear (fear), Surprise (surprise), Disgust (disgust), Frustration (frustration), Confused (confusion), Anxious (anxiety), Startled (startle), and Neutral (neutral).

[0126] Step S500: Detect and resolve conflicts between the coarse-grained labels pre-classified for each modality and the coarse-grained labels predicted by the large model, and iteratively optimize the coarse-grained sentiment classification results through the loss function.

[0127] In this embodiment of the application, step S500 specifically includes the following steps:

[0128] Step S510: For any sampling time Coarse-grained sentiment tags for each modality pre-classification , , And coarse-grained sentiment labels predicted by large models If they are completely identical, they are considered to be conflict-free, and the identical label is directly used as the coarse classification label after conflict resolution. If they are inconsistent, it is determined that there is a conflict.

[0129] in, , , The three single-modal models, namely text, speech, and vision, are respectively in the [number]th [year]. Coarse-grained sentiment pre-classification labels are output in real time.

[0130] Step S520: For cases determined to be conflicting, a weighted voting decision is made using the fusion weights corresponding to each modality to obtain the corresponding sampling time. The coarse classification label for conflict resolution is expressed as follows:

[0131] ,

[0132] in, This represents the weighted sum of the most sentiment categories. For the first At this moment, the first The fusion weights of each modality; This is an indicator function used to identify whether the label category matches; For the first At this moment, the pre-classification coarse-grained sentiment tags for each modality, For a set of modal labels, ; A collection of emotional tags .

[0133] Step S530: Update the model parameters of the coarse classifier and the pre-trained deep learning model using the cross-entropy loss function to achieve iterative optimization of the coarse-grained sentiment classification results. The mathematical expression of the loss function is:

[0134] ,

[0135] in, For the first At this moment, the first Coarse-grained sentiment tags with manual annotations corresponding to each modality. This represents taking the natural logarithm of the confidence probability value.

[0136] Step S600: Integrate fine-grained sentiment tags from different prediction sources, and generate the final fine-grained sentiment tag and the corresponding coarse-grained sentiment tag through a weighted voting mechanism.

[0137] Furthermore, a weighted voting mechanism is used to generate the final fine-grained sentiment tags. for:

[0138] ;

[0139] in, , The voting weights are hyperparameters. For the first Fine-grained sentiment tags for real-time clustering. For the first Fine-grained sentiment tags in the time-major model prediction output;

[0140] Using a mapping function, the final fine-grained sentiment labels are mapped to the final coarse-grained labels. .

[0141] Step S700: Construct a comprehensive evaluation system from multiple dimensions, including multimodal consistency, feature space outlier degree, and conflict resolution decision effect, to quantitatively evaluate the credibility of the final label and identify low-credibility samples.

[0142] In this embodiment of the application, for any time The calculation process and credibility determination process for the multi-dimensional evaluation indicators of the sample are as follows:

[0143] Step S710: Calculate the multimodal consistency score:

[0144] ;

[0145] in, For the first Multimodal consistency score of time-series samples.

[0146] Step S720: Calculate the outlier score in the feature space:

[0147] ;

[0148] in, For the first Outlier score in the feature space of the sample at time step. It is a natural exponential function. The Mahalanobis distance, , , For the fusion feature matrix The corresponding mean and variance; The inverse of the covariance matrix, This represents the transpose of the sample vector and the deviation vector from the center.

[0149] Step S730: Calculate the conflict resolution decision score:

[0150] ;

[0151] in, For the first Conflict resolution decision score for time-series samples This means that when there is no conflict, the weighted voting score for each sentiment category k is calculated, and the maximum value is taken as the decision score at that moment. Indicates when the first Execute this branch when there is no conflict.

[0152] Step S740: After obtaining the scores of these three dimensions, introduce hyperparameter weights to calculate the final overall credibility of the sample:

[0153] ;

[0154] in, Indicates the first The final overall credibility of the time-based samples. For hyperparameter weights, This represents the scores across three dimensions. .

[0155] Step S750: Set a confidence threshold. When the overall confidence of a sample is lower than the threshold, the sample is determined to be a low-confidence sample.

[0156] The credibility threshold is generally set at 80%.

[0157] In an alternative embodiment, low-confidence samples for: .

[0158] By using the output labels and confidence levels of the method of this invention, monitoring personnel can achieve the following: 1) Real-time perception of public opinion pulse, and early warning of escalation of the situation through emotional evolution graphs (such as the transition from "Surprise" to "Angry" to "Fear"); 2) Precisely locate risk sources and identify communities or regions where highly credible "anger" or "fear" content gathers; 3) In-depth evaluation of the effectiveness of the campaign, analyzing the real and complex emotional combinations triggered by the publicity campaign, going beyond simple "positive review rate".

[0159] To fully verify the effectiveness of the method of this invention, two large-scale multimodal opinion sentiment analysis datasets, CMU-MOSI and CMU-MOSEI, which are widely used in the industry, were selected for comprehensive experimental analysis.

[0160] The CMU-MOSI (Multimodal Opinion Sentiment Intensity dataset) contains video clips from 93 movie review topics, totaling 2199 independent speech units.

[0161] CMU-MOSEI (Multimodal Opinion Sentiment and Emotion Intensity dataset) is an upgraded version of it, with a significantly expanded scale of 23,453 video clips, covering approximately 1,000 different speakers and more than 250 diverse topics, and the data distribution is more representative and generalizable.

[0162] Both datasets provide continuous sentiment intensity labels (ranging from -3 to +3), which can be used to derive coarse-grained binary classification (positive / negative) and fine-grained seven-class classification (seven sentiment intensity levels) tasks. To maintain consistency with existing research methods, this experiment will focus on the two classification settings mentioned above and evaluate the model performance accordingly.

[0163] This invention's method is implemented and experimentally verified using Python within the PyCharm integrated development environment. It primarily relies on the following software and tools: Python 3.9+, the PyTorch deep learning framework, Transformers (for BERT, wav2vec 2.0, and large language model calls), the CMU Multimodal SDK (mmsdk), NumPy, scikit-learn, and other scientific computing and evaluation tools. Furthermore, it automates the downloading, parsing, and alignment of the CMU-MOSI / CMU-MOSEI datasets within PyCharm using the official SDK. Through programming, this invention enables a complete automated process within the Python / PyCharm environment, from multimodal data loading, feature extraction, coarse-grained classification, dynamic fusion, automatic invocation of large language model inference, to fine-grained sentiment annotation and performance evaluation.

[0164] This embodiment compares the method of this invention with traditional multimodal fusion methods (such as MMIN and GCNet), comparing key indicators such as binary classification accuracy / F1 score and seven-class classification accuracy of each method on two datasets. The experimental results are shown in Table 1. The references for the MMIN and GCNet methods are: Zhao, J., Li, R., & Jin, Q. (2021). Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities . ACL 2021; Hu, J., Liu, Y., Zhao, J., & Jin, Q. (2021). MMGCN: Multimodal Fusion via Deep Graph Convolution Network for EmotionRecognition in Conversation. ACL 2021.

[0165] Table 1. Comparison of sentiment classification performance of different methods on CMU-MOSI and CMU-MOSEI

[0166]

[0167] As can be seen from Table 1, the method of the present invention is significantly better than the comparative method on both the CMU-MOSI and CMU-MOSEI datasets.

[0168] In binary classification tasks, the present invention improves the accuracy and F1 score by about 4–6 percentage points on both datasets, indicating that the confidence-based multimodal fusion and conflict resolution mechanism can effectively enhance the ability to distinguish sentiment polarity.

[0169] In the seven-class classification task, this invention achieved a more significant performance improvement compared to traditional multimodal fusion methods, verifying the advantages of the coarse-grained to fine-grained progressive reasoning strategy in complex emotion recognition tasks.

[0170] Experimental results show that this invention not only improves the performance of sentiment classification, but also provides an effective technical path for high-reliability automatic labeling of multimodal sentiment data.

[0171] The following is an embodiment of an automatic sentiment data annotation system for fine-grained sentiment analysis according to the present invention, which can be used to execute an embodiment of an automatic sentiment data annotation method for fine-grained sentiment analysis according to the present invention. For details not disclosed in the embodiment of the automatic sentiment data annotation system for fine-grained sentiment analysis according to the present invention, please refer to the embodiment of the automatic sentiment data annotation method for fine-grained sentiment analysis according to the present invention.

[0172] Reference Figure 2 As shown, in one embodiment, an automatic sentiment data annotation system for fine-grained sentiment analysis is provided. This system includes: a multimodal feature extraction module, a coarse-grained sentiment classification module, a feature fusion module, a fine-grained sentiment recognition module, a multi-level voting integration module, a conflict resolution and iterative update module, and a confidence comprehensive evaluation module; wherein,

[0173] The multimodal feature extraction module is used to collect raw multimodal data containing text, speech and visual data, and to extract high-dimensional semantic features of each modality using a pre-trained deep learning model;

[0174] The coarse-grained sentiment classification module is used to perform three-class classification processing (positive, negative, and neutral) based on the high-dimensional semantic features of each modality, and obtain the pre-classified coarse-grained sentiment labels and corresponding classification confidence scores for each modality.

[0175] The feature fusion module is used to dynamically calculate the fusion weight of each modality based on the classification confidence of each modality, and use the fusion weight to perform weighted fusion processing on the high-dimensional semantic features of each modality to generate a fusion feature matrix;

[0176] The fine-grained sentiment recognition module is used to predict fine-grained sentiment tags containing 11 categories based on the fused feature matrix and pre-classified coarse-grained sentiment tags, combined with pre-trained large language model reasoning and spectral clustering methods.

[0177] The conflict resolution and iterative update module is used to detect and resolve conflicts between the coarse-grained labels preclassified by each modality and the coarse-grained labels predicted by the large model, and to iteratively optimize the coarse-grained sentiment classification results through the loss function.

[0178] The multi-level voting integration module is used to integrate fine-grained sentiment tags from different prediction sources, and generate the final fine-grained sentiment tags and corresponding coarse-grained sentiment tags through a weighted voting mechanism.

[0179] The confidence assessment module is used to construct a comprehensive assessment system from multiple dimensions, including multimodal consistency, feature space outlier degree, and conflict resolution decision effect, to quantify the confidence of the final label and identify low-confidence samples.

[0180] The multimodal feature extraction module is connected to the coarse-grained sentiment classification module and the feature fusion module; the coarse-grained sentiment classification module is connected to the feature fusion module, the conflict resolution and iterative update module, and the confidence comprehensive evaluation module; the feature fusion module is connected to the fine-grained sentiment recognition module and the confidence comprehensive evaluation module; the fine-grained sentiment recognition module is connected to the multi-level voting integration module, the conflict resolution and iterative update module, and the confidence comprehensive evaluation module; the conflict resolution and iterative update module is connected to the confidence comprehensive evaluation module.

[0181] Reference Figure 3 As shown, the multimodal feature extraction module includes a text feature extraction submodule, a speech feature extraction submodule, and a visual feature extraction submodule; among which,

[0182] The text feature extraction submodule, including a pre-trained BERT model, takes the input text sequence as input. Encoded as fixed-dimensional text features ;in, , For text feature dimensions;

[0183] The speech feature extraction submodule includes a pre-trained wav2vec 2.0 model, which extracts features from the input audio signal. Extracting speech features ;in, , For speech feature dimensions;

[0184] The visual feature extraction submodule includes Vision Transformer (ViT) or TimeSformer models, corresponding to the extraction of input image or video frame sequences. Extracting spatial and temporal visual features Among them, for images there are For videos , For visual feature dimensions.

[0185] Furthermore, the coarse-grained sentiment classification module includes three independent MLP classifiers: a text coarse classifier, a speech coarse classifier, and a visual coarse classifier; each classifier classifies text features. Speech features Visual features Perform three-class classification and output the corresponding pre-classified coarse-grained sentiment tags. and confidence level ; where the subscript is , , The corresponding different modes.

[0186] Furthermore, three independent multilayer perceptron (MLP) classifiers were pre-trained using a small amount of labeled data to accelerate iterative convergence.

[0187] Furthermore, the feature fusion module includes a weight construction submodule and a weighted fusion submodule. The weight construction submodule is a Softmax layer that performs weighted fusion on the confidence vector. , , Perform Softmax normalization to generate fusion weights for each modality. , , ;

[0188] Fusion weights for each modality:

[0189]

[0190] in, It is a natural exponential function. For temperature parameters, For the first The confidence level probability value of each modality A marker for traversing all modalities; For the mode whose weights are currently being calculated, For each traversed modality.

[0191] The weighted fusion submodule uses the fusion weights of each modality to process text features separately. Speech features Visual features Weighted fusion is performed to obtain the fusion feature matrix. .

[0192] The fine-grained emotion recognition module is connected to the coarse-grained emotion classification module, the multi-level voting integration module, and the conflict resolution and iterative update module, based on a fusion feature matrix. We used a large language model and spectral clustering to obtain fine-grained sentiment label predictions for 11 categories.

[0193] Reference Figure 4As shown, the fine-grained sentiment recognition module includes a cue engineering submodule, a large-model inference submodule, a label parsing and mapping submodule, and a fine-grained sentiment classification module. The cue engineering submodule is connected to the coarse-grained sentiment classification module and the large-model inference submodule, and is used to convert the fused feature matrix into a natural language description vector through a lightweight semantic mapping network, and to pre-classify coarse-grained labels for each modality. , , Mapped to natural language description.

[0194] The large-scale model inference submodule is a large language model, connected to the multi-level voting integration module and the label parsing and mapping submodule. It is used to integrate the above natural language description vectors and coarse-grained sentiment label descriptions based on the preset prompt word templates to construct prompt words that are input to the pre-trained large language model.

[0195] The tag parsing and mapping submodule is used to input prompt words into a pre-trained large language model to obtain coarse- and fine-grained sentiment tags predicted by the large model. ;Establish The mapping function maps the fine-grained sentiment labels predicted by the large model to the coarse-grained labels predicted by the large model. .

[0196] The fine-grained sentiment classification module uses spectral clustering as input, takes a fused feature matrix as input, and outputs a multi-level voting integration module to process the fused feature matrix. Perform spectral clustering to obtain fine-grained sentiment tags with clusters of 11 categories. .

[0197] Furthermore, the conflict resolution and iterative update module is connected to the coarse-grained sentiment classification module and the label parsing and mapping submodule, respectively. It detects and resolves conflicts between different modalities of coarse-grained sentiment labels output by the coarse-grained sentiment classification module and the label parsing and mapping submodule, and updates the state of the next iteration step by optimizing the network parameters of the coarse-grained sentiment classification module through the cross-entropy loss function.

[0198] Furthermore, the confidence assessment module includes a multimodal consistency score calculation submodule, a feature outlier score calculation submodule, a conflict resolution score calculation submodule, and a comprehensive confidence calculation and sample identification submodule; among which,

[0199] The multimodal consistency score calculation submodule is used to calculate the multimodal consistency score.

[0200] Specifically, the consistency score between the final fine-grained sentiment tags and coarse-grained sentiment tags and the fine-grained sentiment tags and coarse-grained sentiment tags of each intermediate stage is used to calculate the multimodal consistency score.

[0201] The feature outlier score calculation submodule is used to calculate the feature space outlier score.

[0202] Specifically, the outlier score of the fusion feature matrix in the overall distribution is calculated based on Mahalanobis distance and used as the outlier score in the feature space.

[0203] The conflict resolution score calculation submodule is used to calculate the conflict resolution decision score.

[0204] Specifically, the conflict resolution effectiveness is scored based on the conflict flags and the weighted voting results during resolution.

[0205] The comprehensive credibility calculation and sample identification submodule is used to calculate the final comprehensive credibility of a sample based on the scores of the above three dimensions, introduce hyperparameter weights, and set a credibility threshold. When the comprehensive credibility of a sample is lower than the threshold, the sample is determined to be a low credibility sample.

[0206] It should be noted that the various functional modules in the embodiments of the present invention can be integrated into one processing module, or each unit can exist as a separate physical entity, or two or more units can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module.

[0207] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A method for automatically labeling sentiment data for fine-grained sentiment analysis, characterized in that, The method includes: Collect raw multimodal data containing text, speech, and visual data, and use pre-trained deep learning models to extract high-dimensional semantic features of each modality; Based on the high-dimensional semantic features of each modality, a three-class classification process of positive, negative and neutral sentiment is performed to obtain the pre-classified coarse-grained sentiment labels and corresponding classification confidence scores for each modality. The fusion weights of each modality are dynamically calculated based on the classification confidence of each modality. The high-dimensional semantic features of each modality are then weighted and fused using the fusion weights to generate a fusion feature matrix. Based on the fusion feature matrix and pre-classified coarse-grained sentiment tags, combined with pre-trained large language model inference and spectral clustering methods, fine-grained sentiment tags containing 11 categories are predicted. The system detects and resolves conflicts between the coarse-grained labels pre-classified by each modality and the coarse-grained labels predicted by the large model, and iteratively optimizes the coarse-grained sentiment classification results through loss function. The process of detecting and resolving conflicts between the coarse-grained labels pre-classified for each modality and the coarse-grained labels predicted by the large model, and iteratively optimizing the coarse-grained sentiment classification results through a loss function, includes: For any sampling time, if the pre-classified coarse-grained sentiment labels of each modality and the coarse-grained sentiment labels predicted by the large model are completely consistent, they are determined to be without conflict, and the consistent labels are directly used as the coarse classification labels after conflict resolution; if they are inconsistent, they are determined to be in conflict. For cases identified as conflicts, a weighted voting decision is made using the fusion weights corresponding to each modality to obtain the coarse classification label for conflict resolution at the corresponding sampling time. The cross-entropy loss function is used to update the model parameters of the coarse classifier and the parameters of the pre-trained deep learning model, thereby achieving iterative optimization of the coarse-grained sentiment classification results. Fine-grained sentiment tags from different prediction sources are integrated, and a weighted voting mechanism is used to generate the final fine-grained sentiment tags, as well as the coarse-grained sentiment tags corresponding to the fine-grained sentiment tags. The final fine-grained sentiment tags are generated through a weighted voting mechanism. for: ； in, This represents the weighted sum of the most sentiment categories. , The voting weights are hyperparameters. For indicator functions, For the first Fine-grained sentiment tags for real-time clustering. For the first Fine-grained sentiment labels in the time-major model prediction output. A collection of emotional tags; Using a mapping function, the final fine-grained sentiment labels are mapped to the final coarse-grained labels; A comprehensive evaluation system is constructed from multiple dimensions, including multimodal consistency, outlier degree in feature space, and conflict resolution decision effect, to quantify the credibility of the final label and identify low-credibility samples. For a sample at any sampling time, the calculation process of its multi-dimensional evaluation indicators and the determination of its credibility are as follows: Calculate the scores for multimodal consistency, feature space outlier, and conflict resolution decision, respectively; After obtaining the scores of these three dimensions, hyperparameter weights are introduced to calculate the final comprehensive credibility of the sample. Set a confidence threshold. When the overall confidence of a sample is lower than the threshold, the sample is determined to be a low-confidence sample.

2. The automatic annotation method for sentiment data for fine-grained sentiment analysis according to claim 1, characterized in that, The extraction of high-dimensional semantic features for each modality using a pre-trained deep learning model includes: Using a pre-trained text parameter feature model, the input text sequence Encoded as fixed-dimensional text features Among them, individual text features BERT is a pre-trained text parameter feature extraction model function. For text sequences The first in Each text unit , The length of the sampling sequence. For text feature dimensions, For dimension The set of real matrices, For length is The set of real column vectors; Using a pre-trained speech parameter feature model, from the input audio signal Extracting speech features Among them, individual speech features , Pre-trained speech feature extraction model function, audio signal The first in One audio signal unit, , For speech feature dimension, For dimension The set of real matrices, For length is The set of real column vectors; Using a pre-trained visual feature model, corresponding to the input image or video frame sequence Extracting spatial and temporal visual features Among them, individual image visual features , For pre-trained image visual feature extraction model functions, single video visual features , For pre-trained video visual feature extraction model functions, For visual feature dimensions, For dimension The set of real matrices, For length is The set of real column vectors, A sequence of image or video frames The first in Each image or video unit .

3. The automatic sentiment data annotation method for fine-grained sentiment analysis according to claim 2, characterized in that, The pre-classified coarse-grained sentiment tags are represented as follows: ， in, For each modality, pre-classify coarse-grained sentiment labels, with subscripts... These correspond to three different modalities: text features, speech features, and visual features. Pos represents positive sentiment, Neg represents negative sentiment, and Neu represents neutral sentiment. To use the sequence length; The classification confidence level is expressed as follows: ， in, For the confidence vector, the subscripts are... These correspond to three different modalities: text features, speech features, and visual features.

4. The automatic annotation method for sentiment data for fine-grained sentiment analysis according to claim 3, characterized in that, The process of dynamically calculating the fusion weights of each modality based on the classification confidence of each modality, and then using these fusion weights to perform weighted fusion processing on the high-dimensional semantic features of each modality to generate a fusion feature matrix includes: For the original confidence vector , , Perform Softmax normalization, subscript These correspond to three different modalities: text features, speech features, and visual features. Temperature parameters are introduced to adjust the smoothness of the weight distribution, and the fusion weights for each mode are set. for: ， in, It is a natural exponential function. For temperature parameters, For the first The confidence level probability value of each modality A marker for traversing all modalities; For the mode whose weights are currently being calculated, For each traversed modality; Based on the calculated fusion weights , , Text features respectively Speech features Visual features Weighted fusion is performed to obtain the fusion feature matrix. The fused feature matrix .

5. The automatic sentiment data annotation method for fine-grained sentiment analysis according to claim 4, characterized in that, The method, based on a fused feature matrix and pre-classified coarse-grained sentiment tags, combined with pre-trained large language model inference and spectral clustering, predicts fine-grained sentiment tags comprising 11 categories, including: The fused feature matrix is ​​converted into a natural language description vector through a lightweight semantic mapping network, while the pre-classified coarse-grained sentiment labels of each modality are mapped to natural language descriptions. Based on the preset prompt word template, natural language description vectors and coarse-grained sentiment label descriptions are integrated to construct prompt words that are input into the pre-trained large language model; Input the prompt words into a pre-trained large language model to obtain fine-grained sentiment labels from the model's predicted output; Establish The mapping function maps the fine-grained sentiment labels predicted by the large model to the coarse-grained sentiment labels predicted by the large model. It refers to the mapping from an 11-dimensional real number space to a 3-dimensional real number space; Spectral clustering was performed on the fused feature matrix to obtain fine-grained sentiment tags containing 11 categories.

6. The automatic annotation method for sentiment data for fine-grained sentiment analysis according to claim 5, characterized in that, The mapping function is defined as follows: ， in, For mapping functions, The input variables for fine-grained sentiment labels are: Happy (pleasure), Angry (anger), Sadness (sadness), Fear (fear), Surprise (surprise), Disgust (disgust), Frustration (frustration), Confused (confusion), Anxious (anxiety), Startled (startle), and Neutral (neutral).

7. The automatic annotation method for sentiment data for fine-grained sentiment analysis according to claim 6, characterized in that, The process of detecting and resolving conflicts between the coarse-grained labels pre-classified for each modality and the coarse-grained labels predicted by the large model, and iteratively optimizing the coarse-grained sentiment classification results through a loss function, includes: For any sampling time Coarse-grained sentiment tags for each modality pre-classification , , And coarse-grained sentiment labels predicted by large models If they are completely consistent, they are determined to be without conflict, and the consistent label is directly used as the coarse classification label after conflict resolution; if they are inconsistent, they are determined to be in conflict. in, , , The three single-modal models, namely text, speech, and vision, are respectively in the [number]th [year]. Real-time output of coarse-grained sentiment pre-classification tags; For cases deemed conflicting, a weighted voting decision is made using the fusion weights corresponding to each modality to obtain the corresponding sampling time. Conflict resolution coarse classification tags Its expression is: ， in, This represents the weighted sum of the most sentiment categories. For the first At this moment, the first The fusion weights of each modality; This is an indicator function used to identify whether the label category matches; For the first At this moment, the pre-classification coarse-grained sentiment tags for each modality, For a set of modal labels, ; A collection of emotional tags ; The cross-entropy loss function is used to update the model parameters of the coarse classifier and the parameters of the pre-trained deep learning model, thereby achieving iterative optimization of the coarse-grained sentiment classification results. The mathematical expression is: ， in, For the first At this moment, the first Coarse-grained sentiment tags with manual annotations corresponding to each modality. This represents taking the natural logarithm of the confidence probability value.

8. The automatic annotation method for sentiment data for fine-grained sentiment analysis according to claim 7, characterized in that, For any sampling time The calculation process and credibility determination process for the multi-dimensional evaluation indicators of the sample are as follows: Calculate the multimodal consistency score: ； in, For the first Multimodal consistency score of samples at time step; Calculate the outlier score in the feature space: ； in, For the first Outlier score in the feature space of the sample at time step. It is a natural exponential function. The Mahalanobis distance, , , These are the fusion feature matrices. The corresponding mean and variance; The inverse of the covariance matrix, This represents the transpose of the sample vector and the deviation vector from the center. Calculate the conflict resolution decision score: ； in, For the first Conflict resolution decision score for time-series samples This means that when there is no conflict, the weighted voting score for each sentiment category k is calculated, and the maximum value is taken as the decision score at that moment. Indicates when the first Execute this branch if there are no conflicts at any given time. After obtaining the scores in these three dimensions, hyperparameter weights are introduced to calculate the final overall credibility of the sample: ； in, Indicates the first The final overall credibility of the time-based samples. For hyperparameter weights, This represents the scores across three dimensions. ; Set a confidence threshold. When the overall confidence of a sample is lower than the threshold, the sample is determined to be a low-confidence sample.

9. An automatic sentiment data annotation system for fine-grained sentiment analysis, characterized in that, The system for performing the method as described in any one of claims 1-8 includes: The multimodal feature extraction module is used to collect raw multimodal data containing text, speech, and visual data, and to extract high-dimensional semantic features of each modality using a pre-trained deep learning model. The coarse-grained sentiment classification module is used to perform three-class classification processing (positive, negative, and neutral) based on the high-dimensional semantic features of each modality, and obtain the pre-classified coarse-grained sentiment labels and corresponding classification confidence scores for each modality. The feature fusion module is used to dynamically calculate the fusion weight of each modality based on the classification confidence of each modality, and use the fusion weight to perform weighted fusion processing on the high-dimensional semantic features of each modality to generate a fusion feature matrix; The fine-grained sentiment recognition module is used to predict fine-grained sentiment tags containing 11 categories based on the fused feature matrix and pre-classified coarse-grained sentiment tags, combined with pre-trained large language model reasoning and spectral clustering methods. The conflict resolution and iterative update module is used to detect and resolve conflicts between the coarse-grained labels preclassified by each modality and the coarse-grained labels predicted by the large model, and to iteratively optimize the coarse-grained sentiment classification results through the loss function. The process of detecting and resolving conflicts between the coarse-grained labels pre-classified for each modality and the coarse-grained labels predicted by the large model, and iteratively optimizing the coarse-grained sentiment classification results through a loss function, includes: For any sampling time, if the pre-classified coarse-grained sentiment labels of each modality and the coarse-grained sentiment labels predicted by the large model are completely consistent, they are determined to be without conflict, and the consistent labels are directly used as the coarse classification labels after conflict resolution; if they are inconsistent, they are determined to be in conflict. For cases identified as conflicts, a weighted voting decision is made using the fusion weights corresponding to each modality to obtain the coarse classification label for conflict resolution at the corresponding sampling time. The cross-entropy loss function is used to update the model parameters of the coarse classifier and the parameters of the pre-trained deep learning model, thereby achieving iterative optimization of the coarse-grained sentiment classification results. The multi-level voting integration module is used to integrate fine-grained sentiment tags from different prediction sources, and generate the final fine-grained sentiment tags and corresponding coarse-grained sentiment tags through a weighted voting mechanism. The final fine-grained sentiment tags are generated through a weighted voting mechanism. for: ； in, This represents the weighted sum of the most sentiment categories. , The voting weights are hyperparameters. For indicator functions, For the first Fine-grained sentiment tags for real-time clustering. For the first Fine-grained sentiment labels in the time-major model prediction output. A collection of emotional tags; Using a mapping function, the final fine-grained sentiment labels are mapped to the final coarse-grained labels; The confidence assessment module is used to construct a comprehensive assessment system from multiple dimensions such as multimodal consistency, feature space outlier degree, and conflict resolution decision effect, to quantify the confidence of the final label and identify low confidence samples. For a sample at any sampling time, the calculation process of its multi-dimensional evaluation indicators and the determination of its credibility are as follows: Calculate the scores for multimodal consistency, feature space outlier, and conflict resolution decision, respectively; After obtaining the scores of these three dimensions, hyperparameter weights are introduced to calculate the final comprehensive credibility of the sample. Set a confidence threshold. When the overall confidence of a sample is lower than the threshold, the sample is determined to be a low-confidence sample.

Images