A multimodal sentiment analysis method based on context retention and sentiment evidence rollback

Abstract

The application discloses a kind of multi-modal sentiment analysis methods based on context maintenance and emotional evidence backtracking, first obtain the image to be analyzed, original text query and emotional guide prompt, then original text query and emotional guide prompt are spliced into coarse-grained query;Then the coarse-grained query and the image to be analyzed are input into large language model to extract and fuse features, output original score, and rely on original score to calculate emotional probability and confidence, and obtain multi-modal emotional candidate set by screening;Again, rely on model coding, cross-modal attention calculates visual token weight, and filters high-weight visual token to construct emotional visual evidence set;Finally, configure first-analysis-after-selection prompt for large language model, execute evidence backtracking mechanism in autoregressive generation, output emotional evidence analysis text and fine-grained emotional label.The application realizes global context maintenance, and improves the accuracy, stability and traceability of fine-grained multi-modal sentiment analysis by using evidence backtracking mechanism.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, computer vision, natural language processing, and sentiment computing, specifically a multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking. Background Technology

[0002] Multimodal large models typically convert input images into visual words and feed these visual words along with the text query into an autoregressive language decoder to output the answer. Fine-grained sentiment understanding differs from single-object recognition: the sentiment conveyed by an image is often determined by the subject's expression, actions, local text, object state, scene background, lighting atmosphere, and the semantic relationships between elements. For example, the same "red" region may represent vitality or anger in a specific context; it is difficult to obtain a stable judgment based solely on local salient regions.

[0003] Existing training methods typically acquire sentiment perception capabilities through supervised fine-tuning or efficient parameter fine-tuning, but these require additional labeled data, training resources, and model weight maintenance. To reduce deployment costs, existing training-free solutions mainly employ cue word enhancement, visual lexical filtering, or latent state intervention during the inference stage. The training-free sentiment perception enhancement method SEPM (Sharpening Emotion Perception in Multimodal Large Language Models) generally performs the following steps: first, it uses coarse-grained sentiment cues to obtain positive or negative polarity; second, it scores visual lexical units based on text-visual attention; then, it removes low-scoring visual lexical units before fine-grained inference to reduce visual redundancy and narrow the range of sentiment candidates; finally, it uses the remaining visual lexical units for sentiment classification. Additionally, some existing sentiment analysis methods request the model to provide analysis through chained cues.

[0004] Current sentiment analysis methods have the following technical problems: (1) The deletion of input-level visual words causes irreversible context loss; low-attention words are regarded as redundant and deleted before inference, but low-scoring areas may still contain scene atmosphere, object relationship or contrast information; since deleted words are inaccessible in subsequent layers, there is a conflict between local denoising and global semantic integrity. (2) The visual basis in the autoregressive language decoder weakens as generation progresses; as the analyzed text becomes longer, the total attention of the autoregressive language decoder to visual words may decrease and gradually rely on language priors. (3) Continuous or unselective visual injection is prone to introducing perturbations; if all visual words are injected in all layers and all time steps, background noise will be reintroduced into the decoder and the delay will be increased; if only a single saliency is continuously enhanced, visual saliency areas unrelated to sentiment may be amplified. (4) The candidate label space does not match the inference difficulty; directly selecting from the complete fine-grained label set will increase confusion between similar sentiments; but prematurely pruning the candidate set with low-confidence coarse polarity will produce irrecoverable polarity errors. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking. Without updating the parameters of the large language model or deleting the complete visual input, it can simultaneously reduce sentiment-irrelevant visual noise, maintain the global context, and improve the accuracy, stability and traceability of fine-grained multimodal sentiment analysis by utilizing the evidence backtracking mechanism.

[0006] The technical solution of this invention is as follows: A multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking specifically includes the following steps: (1) Obtain the image to be analyzed, the original text query and the sentiment guidance prompt. The original text query is used to clarify the specific task requirements for sentiment analysis of the image to be analyzed, and the sentiment guidance prompt is used to constrain the direction of sentiment focus in the process of sentiment analysis of the image to be analyzed. (2) Combine the original text query and sentiment guidance prompts into a coarse-grained query that includes two options: positive sentiment and negative sentiment; (3) Input the coarse-grained query and the image to be analyzed into the autoregressive multimodal large language model, extract the text semantic features and visual features and complete the multimodal feature fusion, output the original score, the original score covers the native global vocabulary built into the autoregressive multimodal large language model, then the original score is normalized to generate the complete vocabulary probability distribution, and then positive and negative sentiment probabilities and confidence are extracted from the complete vocabulary probability distribution. Combine the confidence threshold to filter the original sentiment candidate set and determine the multimodal sentiment candidate set. (4) Input the emotion guidance prompt and the image to be analyzed into the autoregressive multimodal large language model. The image to be analyzed is divided into multiple image sub-blocks. The multiple image sub-blocks are encoded to generate a set of visual words. Then, according to the cross-modal attention mechanism of emotion guidance prompt words pointing to visual words, the importance score of each visual word in the set of visual words is obtained. Finally, multiple visual words with high importance scores are selected to construct the set of emotional visual evidence. (5) Input the image to be analyzed, the original text query, the multimodal sentiment candidate set and the first-analysis-then-selection prompt into the autoregressive multimodal large language model, and execute the evidence backtracking mechanism in the autoregressive generation; the evidence backtracking mechanism is as follows: first calculate the Top-K normalized uncertainty of each target decoding layer in the target decoding layer set under the current generation step, and then determine whether to trigger evidence backtracking according to the uncertainty threshold; after triggering, construct the enhanced feature in the next decoding layer of the target decoding layer, and fuse the constructed enhanced feature with the feedforward network output of the next decoding layer; the model continues autoregressive generation until the end label, and outputs the sentiment evidence analysis text and fine-grained sentiment labels.

[0007] The original set of emotion candidates is denoted as Original sentiment candidate set It includes multiple candidate sentiment tags, which are divided into positive sentiment candidate tags and negative sentiment candidate tags. All positive sentiment candidate tags constitute a positive fine-grained candidate subset. All negative sentiment candidate tags constitute a fine-grained subset of negative candidates. .

[0008] The process of extracting positive and negative sentiment probabilities and confidence levels from the complete vocabulary probability distribution, and then filtering the original sentiment candidate set based on the confidence level threshold to determine the multimodal sentiment candidate set, specifically involves: extracting the probability values ​​corresponding to positive sentiment from the complete vocabulary probability distribution according to the category arrangement order as the positive sentiment probability. Extract the probability value corresponding to negative sentiment as the negative sentiment probability. Then, the confidence level is calculated based on the binary probability distribution consisting of positive and negative sentiment probabilities. Set confidence threshold ,when and In this case, a positive fine-grained candidate subset is selected. As a multimodal emotion candidate set; when and In this case, a negative fine-grained candidate subset is selected. As a multimodal emotion candidate set; when When that happens, the original set of emotion candidates will be... As a candidate set of multimodal sentiments.

[0009] The confidence level is calculated based on a binary probability distribution consisting of positive and negative sentiment probabilities. , see the following formula (1) for details: (1); In equation (1), The probability mean of a binary probability distribution.

[0010] The multiple image sub-blocks are encoded to generate a visual word set, specifically: the multiple image sub-blocks are encoded by the visual encoder of an autoregressive multimodal large language model to obtain a visual feature sequence. ,in, Represents the total number of image sub-blocks. The dimension of the feature vector corresponding to a single image sub-patch; visual feature sequence Then, it is mapped to a visual lexical set through the visual-linguistic projection layer of the autoregressive multimodal large language model. , ,in, The unified embedding vector dimension representing a single visual word unit. Represents a single visual word in a set of visual words.

[0011] The method involves using a cross-modal attention mechanism based on the emotional guidance prompt word element pointing to the visual word element to calculate the importance score of each visual word element in the visual word element set. Finally, multiple visual word elements with high importance scores are selected to construct an emotional visual evidence set. Specifically, the attention sub-matrix A of the emotional guidance prompt pointing to the visual word element set is extracted from the last layer of the autoregressive multimodal large language model's visual text interaction attention layer. Then, the importance score of each visual word element is calculated using the following formula (2). : (2); In equation (2), The first in the visual word set Importance score of each visual word element; This represents the total number of attention heads in the current layer of the autoregressive multimodal large language model's multi-head attention structure. The total number of text elements after the emotional guidance prompts are broken down; Representing the Under the attention of each person, the first emotional guidance prompt The text terminology points to the first... The attention weights of each visual word are elements in the attention submatrix A; Importance scores of all visual lexicons in the visual lexicon set Sort from low to high, and then sort the results from the previous results. Each visual lexical unit is assigned to the redundant set. Then, the remaining items not included in the redundant set are... Visual morphemes are classified into the set of emotional visual evidence. , .

[0012] The calculation of the Top-K normalized uncertainty of each target decoding layer in the target decoding layer set at the current generation step, and then determining whether to trigger evidence backtracking based on the uncertainty threshold, is as follows: First, a set of target decoding layers is pre-defined or adaptively selected in the autoregressive multimodal large language model. Inside, for the current generation step and target decoding layer , target decoding layer Hidden state The mapping is a complete probability distribution, as shown in equation (3) below: (3); In equation (3), the target decoding layer Belongs to the target decoding layer set , The target set of decoding layers represents the total number of decoding layers in an autoregressive multimodal large language model. It is a continuous interval selected from all decoding layers of the autoregressive multimodal large language model; The output projection weight matrix represents the autoregressive multimodal large language model; This represents the softmax normalization function; Represents the current generation step Next, target decoding layer The corresponding complete probability distribution; Then The probability of each word element is sorted from highest to lowest, and the word with the highest probability is selected. A set of high-probability candidate lexical units. , see the following formula (4) for details: (4); In equation (4), The probability of each word element is the probability of each word element in the built-in native global vocabulary of the autoregressive multimodal large language model; It represents the total number of candidate lexical units in the set of high-probability candidate lexical units; Then, for the set of high-probability candidate lexical units... The probability of each candidate word element is normalized, as shown in the following formula (5): (5); In equation (5), Represents the first in the set of high-probability candidate lexical units The probability of each candidate word; This represents the probability of traversing all candidate words in the set of high-probability candidate words; Representing the The normalized probability of each candidate term; Finally, the Top-K normalized uncertainty is calculated. , see the following formula (6) for details: (6); In equation (6), It represents the total number of candidate lexical units in the set of high-probability candidate lexical units; Represents the current generation step Next, target decoding layer Top-K normalized uncertainty; When the target decoding layer of Not less than the uncertainty threshold When this happens, evidence backtracking is triggered; when the target decoding layer... of Less than the uncertainty threshold In this case, evidence backtracking will not be triggered, and the original decoding process will be maintained.

[0013] After the triggering, an enhanced feature is constructed in the next decoding layer of the target decoding layer, specifically as follows: When the target decoding layer When evidence backtracking is triggered, the first Decoding layer current generation step Hidden state As a query, the set of emotional visual evidence Linear projection is the key set Sum value set The enhanced features are calculated using the following formula (7). : (7); In equation (7), Represents hidden state Set of keys single key vector The similarity between them is calculated using inner product, cosine similarity, or attention scoring, and represents the hidden state. and emotional visual evidence set The degree of matching and fit between individual visual morphemes; Representative value set Zhongyu A one-to-one correspondence of value vectors; This represents a gated activation function that adaptively adjusts the injection weights of different visual terms based on their similarity.

[0014] The process of fusing the constructed enhanced features with the feedforward network output of the next decoding layer specifically involves: The first Decoding layer current generation step Hidden state The signal is fed into the decoder's built-in feedforward network, which then outputs a native feedforward output. , see the following formula (8) for details: (8); In equation (8), Represents a feedforward network; Then the native feedforward output With enhanced features Perform dimension alignment and according to the injection ratio. The fusion is shown in the following formula (9): (9); In equation (9), The newly generated feedforward output will be fused. Replace native feedforward output Access the decoder's subsequent residual connection link to complete the first step. Evidence enhancement update at the decoding layer.

[0015] Based on the "analyze first, select later" prompt, an analysis field and a selection field are set in the autoregressive generation process of the autoregressive multimodal large language model. In the analysis field, sentiment evidence analysis text corresponding to the image content is generated, and in the selection field, fine-grained sentiment tags are selected from the multimodal sentiment candidate set based on the sentiment evidence analysis text. For each generation step in the two fields, forward operations of all decoding layers are performed, and an evidence backtracking mechanism is performed in the target decoding layer.

[0016] Advantages of this invention: (1) The present invention changes the purpose of visual evidence screening from "deleting input" to "constructing an emotional visual evidence set as an auxiliary memory"; the complete image to be analyzed still participates in the entire autoregressive generation process under the prompt of analysis and selection, so that the background, object relationship and scene atmosphere are always accessible; the screened emotional visual evidence only updates the feedforward network output of the next decoding layer when the evidence backtracking is triggered, so that the complete context and the denoised evidence complement each other.

[0017] (2) This invention relies on the optimization mechanism of evidence backtracking and adaptive decoding screening to make the autoregressive multimodal large language model more reliable in mining and judging the deep and subtle emotional features of the image to be analyzed, and solves the problem of insufficient fine-grained emotional discrimination of existing sentiment analysis models.

[0018] (3) In the evidence backtracking mechanism of the present invention, the attention of all visual words is not increased indiscriminately. Instead, a precise and targeted enhancement method is adopted to only improve the attention convergence of effective emotional visual evidence words, thus avoiding meaningless global attention enhancement.

[0019] (4) This invention does not change the original parameters of the autoregressive multimodal large language model, does not require additional manually labeled emotional data, and does not require special fine-tuning for image emotion recognition tasks. It can be directly put into use based on the existing pre-trained autoregressive multimodal large language model, which greatly reduces the manpower and computing power costs of data production and model training. Compared with the existing autoregressive multimodal large language model, the additional operations are only concentrated in the construction of the emotional visual evidence set, uncertainty trigger judgment and visual evidence enhancement fusion stage. Moreover, the evidence backtracking is only selectively triggered in the decoding layer of a small number of high uncertainty targets, and the peak memory increase in the reasoning process is small, which is convenient for engineering deployment in a conventional computing power environment.

[0020] (5) This invention sets up a first-analysis-then-selection prompt for the autoregressive multimodal large language model. In the autoregressive generation process, visual facts and emotional evidence are output first, and then fine-grained emotional tags are selected from the multimodal emotional candidate set. Combined with the evidence backtracking mechanism of the target decoding layer, the emotional judgment has traceable visual evidence and analysis process, and the interpretability of the results is improved. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] See Figure 1 A multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking specifically includes the following steps: (1) Obtain the image to be analyzed, the original text query (which of the following descriptions can represent the emotion of the image?), and the emotion guidance prompt (please pay attention to the main emotion conveyed by the image); the emotion guidance prompt is a preset template built into the autoregressive multimodal large language model, and the image to be analyzed and the original text query are user input; (2) Divide the image to be analyzed into multiple image sub-blocks, and simultaneously combine the original text query and sentiment guidance prompts into a coarse-grained query containing two options: positive sentiment and negative sentiment. The coarse-grained query is: Please pay attention to the main sentiment conveyed by the image; Which of the following descriptions can represent the sentiment of the image? A. < Positive fine-grained candidate subset> >; B.< Negative fine-grained candidate subset Output only A or B; (3) Constructing a multimodal sentiment candidate set, which specifically includes the following steps: S31. Input the coarse-grained query and the image to be analyzed into the autoregressive multimodal large language model, extract the text semantic features and visual features and complete the multimodal feature fusion, output the raw score, the raw score covers the native global vocabulary built into the autoregressive multimodal large language model, and then the raw score is normalized to generate the complete vocabulary probability distribution. S32. Based on the category arrangement order, extract the probability values ​​corresponding to positive sentiment from the probability distribution of the complete vocabulary as the positive sentiment probability. Extract the probability value corresponding to negative sentiment as the negative sentiment probability. Then, the confidence level is calculated based on the binary probability distribution consisting of positive and negative sentiment probabilities. , see the following formula (1) for details: (1); In equation (1), The probability mean of a binary probability distribution; confidence level. The calculation is not limited to the variance calculation in equation (1), but can also use the maximum probability difference, prediction entropy, energy fraction or calibrated confidence level; S33, Then set the confidence threshold. Confidence threshold The value ranges from 0.00 to 1.00, with 0.12 being preferred, representing the confidence threshold. The determination is adaptively based on the validation set, candidate label distribution, or online calibration results; Configure the original sentiment candidate set Original sentiment candidate set It includes multiple candidate emotion tags, such as anger, awe, satisfaction, disgust, excitement, fear, pleasure, sadness, and surprise. These candidate emotion tags are divided into positive and negative emotion candidate tags, with all positive emotion candidate tags forming a positive fine-grained candidate subset. All negative sentiment candidate tags constitute a fine-grained subset of negative candidates. ; when and In this case, a positive fine-grained candidate subset is selected. As a multimodal emotion candidate set; when and In this case, a negative fine-grained candidate subset is selected. As a multimodal emotion candidate set; when When that happens, the original set of emotion candidates will be... As a multimodal sentiment candidate set, it reduces the risk of irreversible candidate exclusion caused by low-confidence coarse-grained judgment; (4) Input the emotion guidance cues (please pay attention to the main emotion conveyed by the image) and the image to be analyzed into the autoregressive multimodal large language model to construct an emotion visual evidence set, specifically: S41. The image to be analyzed is segmented into multiple image sub-blocks. These sub-blocks are then encoded to generate a visual word set. Specifically, the multiple image sub-blocks are encoded by the visual encoder of an autoregressive multimodal large language model to obtain a visual feature sequence. ,in, Represents the total number of image sub-blocks. The dimension of the feature vector corresponding to a single image sub-patch; visual feature sequence Then, it is mapped to a visual lexical set through the visual-linguistic projection layer of the autoregressive multimodal large language model. , ,in, The unified embedding vector dimension representing a single visual word unit. Represents a single visual word in a set of visual words; S42. Extract the attention submatrix A of the set of visual words that point to the sentiment guidance cues from the last layer of the autoregressive multimodal large language model, the visual text interaction attention layer. Then, calculate the importance score of each visual word using the following formula (2). : (2); In equation (2), The first in the visual word set Importance score of each visual word element; This represents the total number of attention heads in the current layer of the autoregressive multimodal large language model's multi-head attention structure. The total number of text elements after the emotional guidance prompts are broken down; Representing the Under the attention of each person, the first emotional guidance prompt The text terminology points to the first... The attention weights of each visual word are elements in the attention submatrix A; Importance scores of all visual lexicons in the visual lexicon set Sort from lowest to highest, and then sort the results from the previous results. Each visual lexical unit is assigned to the redundant set. , The value should be between 0.1 and 0.5, with 0.3 being preferred; then the remaining values ​​not included in the redundant set should be removed. Visual morphemes are classified into the set of emotional visual evidence. , ; (5) Input the image to be analyzed, the original text query, the multimodal sentiment candidate set and the first-analysis-then-selection prompt into the autoregressive multimodal large language model, and execute the evidence backtracking mechanism in the autoregressive generation; The evidence retrospection mechanism is as follows: S51. First, in the autoregressive multimodal large language model, a target decoding layer set is pre-set or adaptively selected. Inside, for the current generation step and target decoding layer , target decoding layer Hidden state The mapping is a complete probability distribution, as shown in equation (3) below: (3); In equation (3), the target decoding layer Belongs to the target decoding layer set , The target set of decoding layers represents the total number of decoding layers in an autoregressive multimodal large language model. It is a continuous interval selected from all decoding layers of the autoregressive multimodal large language model; in the LLaVA-1.5-7B embodiment, when using code indexing starting from 0, =11、 =20; The output projection weight matrix represents the autoregressive multimodal large language model; This represents the softmax normalization function; Represents the current generation step Next, target decoding layer The corresponding complete probability distribution; S52, will The probability of each word element is sorted from highest to lowest, and the word with the highest probability is selected. A set of high-probability candidate lexical units. , see the following formula (4) for details: (4); In equation (4), The probability of each word element is the probability of each word element in the built-in native global vocabulary of the autoregressive multimodal large language model; This represents the total number of candidate lexical units in the set of high-probability candidate lexical units. ; S53, For high-probability candidate word sets The probability of each candidate word element is normalized, as shown in the following formula (5): (5); In equation (5), Represents the first in the set of high-probability candidate lexical units The probability of each candidate word; This represents the probability of traversing all candidate words in the set of high-probability candidate words; Representing the The normalized probability of each candidate term; S54. Calculate the Top-K normalized uncertainty. , see the following formula (6) for details: (6); In equation (6), It represents the total number of candidate lexical units in the set of high-probability candidate lexical units; Represents the current generation step Next, target decoding layer Top-K normalized uncertainty; S55, when the target decoding layer of Not less than the uncertainty threshold When this happens, evidence backtracking is triggered; when the target decoding layer... of Less than the uncertainty threshold If the uncertainty threshold is not met, evidence backtracking will not be triggered, and the original decoding process will continue; The value range is 0.6 to 0.7, with 0.7 being preferred; S56, When the target decoding layer When evidence backtracking is triggered, the first Decoding layer current generation step Hidden state As a query, the set of emotional visual evidence Linear projection is the key set Sum value set The enhanced features are calculated using the following formula (7). : (7); In equation (7), Represents hidden state Set of keys single key vector The similarity between them is calculated using inner product, cosine similarity, or attention scoring, and represents the hidden state. and emotional visual evidence set The degree of matching and fit between individual visual morphemes; Representative value set Zhongyu A one-to-one correspondence of value vectors; This represents a gated activation function that adaptively adjusts the injection weights of different visual words based on their similarity. S57, the first Decoding layer current generation step Hidden state The signal is fed into the decoder's built-in feedforward network, which then outputs a native feedforward output. , see the following formula (8) for details: (8); In equation (8), Represents a feedforward network; Then the native feedforward output With enhanced features Perform dimension alignment and adjust according to the injection ratio. The fusion is shown in the following formula (9): (9); In the formula, , preferably 0.1; the newly generated feedforward output will be fused Replace native feedforward output Access the decoder's subsequent residual connection link to complete the first step. Evidence enhancement updates at the decoding layer; Based on the "analyze first, select later" prompt, analysis fields and selection fields are set in the autoregressive generation process of the autoregressive multimodal large language model. Sentiment evidence analysis text corresponding to the image content is generated in the analysis field, and fine-grained sentiment tags are selected from the multimodal sentiment candidate set based on the sentiment evidence analysis text in the selection field. For each generation step in the two fields, forward operations of all decoding layers are performed, and an evidence backtracking mechanism is performed in the target decoding layer.

[0024] Performance Analysis: I. Four control ablation experimental groups were set up. The first group used filtered image input + filtered emotional visual evidence backtracking; the second group used filtered image input + all emotional visual evidence backtracking; the third group used complete image input + filtered emotional visual evidence backtracking; and the fourth group used complete image input + all emotional visual evidence backtracking. Filtered image input means replacing the complete image input with the constructed emotional visual evidence set in the autoregressive multimodal large language model for decoding. Complete image input means inputting the complete image into the autoregressive multimodal large language model for decoding. Filtered emotional visual evidence backtracking involves using the constructed emotional visual evidence set to build enhanced features to complete the visual evidence backtracking. All emotional visual evidence backtracking involves using all visual lexical units in the visual lexical set to build enhanced features to complete the visual evidence backtracking. The prediction accuracy of the four control ablation experimental groups for sentiment analysis is shown in Table 1 below. Table 1

[0025] Table 1 To be included in the redundant set The proportion of visual lexical units.

[0026] As shown in Table 1, the prediction results of the third experimental group (the present invention) are superior to the other three groups. It takes into account both global context constraints and precise directional feature reinforcement, avoids local information loss and invalid visual noise interference, and finally achieves the best multimodal sentiment reasoning accuracy, which verifies the rationality and effectiveness of the architecture design.

[0027] II. Fine-grained emotion recognition analysis: On five image sentiment datasets—Emotion6, EmoSet, WebEmo7, WebEmo25, and ArtPhoto—LLaVA-1.5-7B, LLaVA-1.5-13B, and Qwen2.5-VL-7B were used as autoregressive multimodal large language models, and the SEPM (Sharpening Emotion Perception in Multimodal Large Language Models) emotion perception enhancement algorithm for multimodal large models was used as a contrastive sentiment analysis method. The first group used the LLaVA-1.5-7B autoregressive multimodal language model and the SEPM sentiment analysis method; the second group used the LLaVA-1.5-7B autoregressive multimodal language model and the method of this invention; the third group used the LLaVA-1.5-13B autoregressive multimodal language model and the SEPM sentiment analysis method; the fourth group used the LLaVA-1.5-13B autoregressive multimodal language model and the method of this invention; the fifth group used the Qwen2.5-VL-7B autoregressive multimodal language model and the SEPM sentiment analysis method; and the sixth group used the Qwen2.5-VL-7B autoregressive multimodal language model and the method of this invention. The average accuracy of sentiment analysis for the six control groups on five image sentiment datasets is shown in Table 2 below. Table 2

[0028] As shown in Table 2, the average accuracy of the second group increased from 42.91% to 45.08%, a 2.17 percentage point improvement. The fourth group also achieved a 2.70 percentage point improvement in accuracy on the LLaVA-1.5-13B model compared to the third group, and the sixth group achieved a 1.52 percentage point improvement in accuracy on the Qwen2.5-VL-7B model compared to the fifth group. This demonstrates the effectiveness and model versatility of the method of this invention in improving the accuracy of fine-grained image emotion recognition.

[0029] III. Mechanism Analysis of Evidence Retrospection Mechanism: The original autoregressive multimodal large language model (baseline) and the autoregressive multimodal large language model using the method of this invention were compared at different decoding steps in terms of visual attention quality and evidence grounding ratio. Visual attention quality (VM) is used to characterize the total amount of attention allocated to all visual words; evidence grounding ratio (EGR) is used to characterize the proportion of visual attention that falls on emotional visual evidence words. The comparison results are shown in Table 3 below: Table 3

[0030] As shown in Table 3, the baseline visual attention quality decreased from 0.1427 in step 16 to 0.0412 in step 96. The visual attention quality of this invention does not indiscriminately improve all visual attention, but rather enhances the concentration of emotional visual evidence in the middle and later stages: as decoding progresses into the middle and later stages, the baseline EGR declines significantly, while the EGR of this invention remains higher than the baseline, especially with an improvement of +0.0269 in step 96. This indicates that the evidence backtracking mechanism directs limited visual attention to key visual words that truly carry emotional information in the middle and later stages of decoding, significantly increasing the attention ratio of effective emotional visual evidence words.

[0031] IV. Performance overhead analysis of engineering deployment: This invention does not update the parameters of the autoregressive multimodal large language model and does not require additional sentiment annotation or task fine-tuning. The additional overhead mainly comes from the construction of the sentiment visual evidence set, uncertainty trigger determination, and visual evidence enhancement fusion stages. Since the evidence backtracking mechanism is only executed at the target decoding layer and when there is high uncertainty, the peak memory increase is small. The engineering deployment performance and hardware resource overhead of this invention in the inference stage are compared with the baseline. The comparison results are shown in Table 4 below: Table 4

[0032] In Table 4, the generation time (TPOT) for each lexical unit refers to the average inference time taken by the model to generate a single lexical unit corresponding to a fine-grained sentiment tag; throughput is the number of lexical units generated per second; and peak video memory is the peak video memory for inference of the graphics card hardware.

[0033] As shown in Table 4, the computation time, processing speed, and incremental increase in hardware resource consumption of the inference process of this invention are very low. It does not require high computing power and hardware costs, and the inference overhead is controllable and has strong engineering applicability.

[0034] V. Evaluation Indicator Analysis: Evaluation analysis was conducted on the ArtEmis open sentiment understanding evaluation dataset. The results of the evaluation metrics are shown in Table 5 below. ACC represents the label prediction accuracy; F1 represents the F1 score, which is the harmonic mean of precision (P) and recall (R); the review score is a quality score given by a third-party large model to the generated natural language sentiment description text, evaluating the quality of open-source generated content from humanistic perspectives such as visual foundation, sentiment consistency, and logical rationality. Table 5

[0035] As shown in Table 5, the ACC, F1 and review scores of the complete method of this invention are higher than those of SEPM and also higher than those of the method without evidence backtracking mechanism, indicating that evidence backtracking not only affects the final tag words, but also affects the longer intermediate sentiment analysis trajectory.

[0036] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking, characterized by: Specifically, it includes the following steps: (1) Obtain the image to be analyzed, the original text query and the sentiment guidance prompt. The original text query is used to clarify the specific task requirements for sentiment analysis of the image to be analyzed, and the sentiment guidance prompt is used to constrain the direction of sentiment focus in the process of sentiment analysis of the image to be analyzed. (2) Combine the original text query and sentiment guidance prompts into a coarse-grained query that includes two options: positive sentiment and negative sentiment; (3) Input the coarse-grained query and the image to be analyzed into the autoregressive multimodal large language model, extract the text semantic features and visual features and complete the multimodal feature fusion, output the original score, the original score covers the native global vocabulary built into the autoregressive multimodal large language model, then the original score is normalized to generate the complete vocabulary probability distribution, and then positive and negative sentiment probabilities and confidence are extracted from the complete vocabulary probability distribution. Combine the confidence threshold to filter the original sentiment candidate set and determine the multimodal sentiment candidate set. (4) Input the emotion guidance prompt and the image to be analyzed into the autoregressive multimodal large language model. The image to be analyzed is divided into multiple image sub-blocks. The multiple image sub-blocks are encoded to generate a set of visual words. Then, according to the cross-modal attention mechanism of emotion guidance prompt words pointing to visual words, the importance score of each visual word in the set of visual words is obtained. Finally, multiple visual words with high importance scores are selected to construct the set of emotional visual evidence. (5) Input the image to be analyzed, the original text query, the multimodal sentiment candidate set and the first-analysis-then-selection prompt into the autoregressive multimodal large language model, and execute the evidence backtracking mechanism in the autoregressive generation; the evidence backtracking mechanism is specifically as follows: first calculate the Top-K normalized uncertainty of each target decoding layer in the target decoding layer set under the current generation step, and then determine whether to trigger evidence backtracking based on the uncertainty threshold; After triggering, enhanced features are constructed in the next decoding layer of the target decoding layer, and the constructed enhanced features are fused with the feedforward network output of the next decoding layer; the model continues to generate autoregressive labels until the end, and outputs sentiment evidence analysis text and fine-grained sentiment labels.

2. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 1, characterized in that: The original set of emotion candidates is denoted as Original sentiment candidate set It includes multiple candidate sentiment tags, which are divided into positive sentiment candidate tags and negative sentiment candidate tags. All positive sentiment candidate tags constitute a positive fine-grained candidate subset. All negative sentiment candidate tags constitute a fine-grained subset of negative candidates. .

3. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 2, characterized in that: The process of extracting positive and negative sentiment probabilities and confidence levels from the complete vocabulary probability distribution, and then filtering the original sentiment candidate set based on the confidence level threshold to determine the multimodal sentiment candidate set, specifically involves: extracting the probability values ​​corresponding to positive sentiment from the complete vocabulary probability distribution according to the category arrangement order as the positive sentiment probability. Extract the probability value corresponding to negative sentiment as the negative sentiment probability. Then, the confidence level is calculated based on the binary probability distribution consisting of positive and negative sentiment probabilities. Set confidence threshold ,when and In this case, a positive fine-grained candidate subset is selected. As a multimodal emotion candidate set; when and In this case, a negative fine-grained candidate subset is selected. As a multimodal emotion candidate set; when When that happens, the original set of emotion candidates will be... As a candidate set of multimodal sentiments.

4. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 3, characterized in that: The confidence level is calculated based on a binary probability distribution consisting of positive and negative sentiment probabilities. , see the following formula (1) for details: （1）； In equation (1), The probability mean of a binary probability distribution.

5. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 3, characterized in that: The multiple image sub-blocks are encoded to generate a visual word set, specifically: the multiple image sub-blocks are encoded by the visual encoder of an autoregressive multimodal large language model to obtain a visual feature sequence. ,in, Represents the total number of image sub-blocks. The dimension of the feature vector corresponding to a single image sub-patch; visual feature sequence Then, it is mapped to a visual lexical set through the visual-linguistic projection layer of the autoregressive multimodal large language model. , ,in, The unified embedding vector dimension representing a single visual word unit. Represents a single visual word in a set of visual words.

6. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 5, characterized in that: The method involves using a cross-modal attention mechanism based on the emotional guidance prompt word element pointing to the visual word element to calculate the importance score of each visual word element in the visual word element set. Finally, multiple visual word elements with high importance scores are selected to construct an emotional visual evidence set. Specifically, the attention sub-matrix A of the emotional guidance prompt pointing to the visual word element set is extracted from the last layer of the autoregressive multimodal large language model's visual text interaction attention layer. Then, the importance score of each visual word element is calculated using the following formula (2). : （2）； In equation (2), The first in the visual word set Importance score of each visual word element; This represents the total number of attention heads in the current layer of the autoregressive multimodal large language model's multi-head attention structure. The total number of text elements after the emotional guidance prompts are broken down; Representing the Under the attention of each person, the first emotional guidance prompt The text terminology points to the first... The attention weights of each visual word are elements in the attention submatrix A; Importance scores of all visual lexicons in the visual lexicon set Sort from low to high, and then sort the results from the previous results. Each visual lexical unit is assigned to the redundant set. Then, the remaining items not included in the redundant set are... Visual morphemes are classified into the set of emotional visual evidence. , .

7. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 6, characterized in that: The calculation of the Top-K normalized uncertainty of each target decoding layer in the target decoding layer set at the current generation step, and then determining whether to trigger evidence backtracking based on the uncertainty threshold, is as follows: First, a set of target decoding layers is pre-defined or adaptively selected in the autoregressive multimodal large language model. Inside, for the current generation step and target decoding layer , target decoding layer Hidden state The mapping is a complete probability distribution, as shown in equation (3) below: （3）； In equation (3), the target decoding layer Belongs to the target decoding layer set , The target set of decoding layers represents the total number of decoding layers in an autoregressive multimodal large language model. It is a continuous interval selected from all decoding layers of the autoregressive multimodal large language model; The output projection weight matrix represents the autoregressive multimodal large language model; This represents the softmax normalization function; Represents the current generation step Next, target decoding layer The corresponding complete probability distribution; Then The probability of each word element is sorted from highest to lowest, and the word with the highest probability is selected. A set of high-probability candidate lexical units. , see the following formula (4) for details: （4）； In equation (4), The probability of each word element is the probability of each word element in the built-in native global vocabulary of the autoregressive multimodal large language model; It represents the total number of candidate lexical units in the set of high-probability candidate lexical units; Then, for the set of high-probability candidate lexical units... The probability of each candidate word element is normalized, as shown in the following formula (5): （5）； In equation (5), Represents the first high-probability candidate word group in the set. The probability of each candidate word; This represents the probability of traversing all candidate words in the set of high-probability candidate words; Representing the The normalized probability of each candidate term; Finally, the Top-K normalized uncertainty is calculated. , see the following formula (6) for details: （6）； In equation (6), It represents the total number of candidate lexical units in the set of high-probability candidate lexical units; Represents the current generation step Next, target decoding layer Top-K normalized uncertainty; When the target decoding layer of Not less than the uncertainty threshold When this happens, evidence backtracking is triggered; when the target decoding layer... of Less than the uncertainty threshold In this case, evidence backtracking will not be triggered, and the original decoding process will be maintained.

8. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 7, characterized in that: After the triggering, an enhanced feature is constructed in the next decoding layer of the target decoding layer, specifically as follows: When the target decoding layer When evidence backtracking is triggered, the first Decoding layer current generation step Hidden state As a query, the set of emotional visual evidence Linear projection is the key set Sum value set The enhanced features are calculated using the following formula (7). : （7）； In equation (7), Represents hidden state Set of keys single key vector The similarity between them is calculated using inner product, cosine similarity, or attention scoring, and represents the hidden state. and emotional visual evidence set The degree of matching and fit between individual visual morphemes; Representative value set Zhongyu A one-to-one correspondence of value vectors; This represents a gated activation function that adaptively adjusts the injection weights of different visual terms based on their similarity.

9. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 8, characterized in that: The process of fusing the constructed enhanced features with the feedforward network output of the next decoding layer specifically involves: The first Decoding layer current generation step Hidden state The signal is fed into the decoder's built-in feedforward network, which then outputs a native feedforward output. , see the following formula (8) for details: （8）； In equation (8), Represents a feedforward network; Then the native feedforward output With enhanced features Perform dimension alignment and according to the injection ratio. The fusion is shown in the following formula (9): （9）； In equation (9), The newly generated feedforward output will be fused. Replace native feedforward output Access the decoder's subsequent residual connection link to complete the first step. Evidence enhancement update at the decoding layer.

10. The multimodal sentiment analysis method based on context preservation and sentiment evidence backtracking according to claim 1, characterized in that: Based on the "analyze first, select later" prompt, an analysis field and a selection field are set in the autoregressive generation process of the autoregressive multimodal large language model. In the analysis field, sentiment evidence analysis text corresponding to the image content is generated, and in the selection field, fine-grained sentiment tags are selected from the multimodal sentiment candidate set based on the sentiment evidence analysis text. For each generation step in the two fields, forward operations of all decoding layers are performed, and an evidence backtracking mechanism is performed in the target decoding layer.

Images