Emotion classification method based on visual language model and conditional reasoning

Abstract

The invention provides a sentiment classification method based on a visual language model and conditional reasoning, which comprises the following steps of: firstly, acquiring a text picture pair and labeling sentiment labels on the text picture pair to form a sentiment label set; then, a visual language model is used as a strategy model, general reasoning and conditional reasoning are carried out on the text picture pairs respectively, reasoning characterization, emotion prediction labels and conditional reasoning results are generated, and response samples are formed after the reasoning characterization, the emotion prediction labels and the conditional reasoning results are combined; and calculating a reward value and an advantage estimation value based on the response sample to optimize the strategy model, finally utilizing the optimized strategy model to carry out sentiment prediction on a new text picture pair, and outputting a final sentiment classification result. According to the method, the defect that a traditional multi-classification model is easily interfered by noise texts or complex visual contents is overcome, the classification precision is improved, a general reasoning process and a conditional reasoning process of a strategy model are recombined into a group of response samples, it is guaranteed that each group of response contains different classification labels, and the problem of advantage collapse is solved.

Description

Technical Field

[0001] This invention relates to sentiment classification methods in the fields of computer vision and natural language processing, specifically to a sentiment classification method based on visual language models and conditional reasoning. Background Technology

[0002] In recent years, sentiment analysis has received widespread attention due to its significant applications in e-commerce, intelligent customer service, and online public opinion monitoring. Traditional methods mostly rely on a single modality (such as text), but humans often use multiple signals simultaneously, including text, facial expressions, and speech, when expressing emotions. Therefore, multimodal sentiment analysis has gradually become a research hotspot. Compared to single-modal methods, multimodal methods can better capture complementary and interactive information between different modalities, thereby improving the accuracy of sentiment recognition. However, a core challenge of multimodal sentiment analysis lies in how to effectively integrate and model intramodal and intermodal information.

[0003] Thanks to the rapid development of Vision-Language Models (VLMs) in recent years, the long-standing challenge of cross-modal sentiment analysis has found a new solution. Traditional unimodal methods often only capture local features in images or text, making it difficult to fully understand the semantic relationships between the two. The emergence of VLMs allows models to simultaneously model image and text information in a unified semantic space, thereby achieving cross-modal alignment and fusion understanding. Specifically, VLMs typically employ a dual-stream or fusion-based architecture, deeply integrating visual feature extraction networks (such as ViT, CLIP, and ResNet) with language models (such as Transformer or large-scale language models LLM). During large-scale cross-modal pre-training, the model learns rich semantic mapping relationships between images and text by performing various tasks—image-text matching, image captioning, and visual question answering (VQA). This large-scale pre-training not only empowers the model to understand unimodal content, but more importantly, it cultivates its ability to perform semantic reasoning, sentiment recognition, and knowledge transfer under multimodal input.

[0004] However, directly applying VLM to multimodal sentiment classification tasks still has the following problems:

[0005] On the one hand, VLMs often exhibit deterministic outputs during the generative inference phase, with small differences between different sampling results. This makes it difficult for reward signals to form effective distinctions during optimization based on Verifiable Reward Reinforcement Learning (VRRL). On the other hand, in sentiment classification tasks, reward signals typically exist in a binary form of "correct or incorrect," exhibiting significant sparsity. When multiple sampling predictions all yield the same reward, the advantage estimate tends to zero, leading to "advantage collapse" and preventing the model from obtaining effective gradient updates.

[0006] This problem not only slows down the convergence speed and reduces the stability of the model during training, but also directly affects the accuracy of sentiment classification results. Specifically, the model has difficulty distinguishing semantically similar sentiment categories (such as "pleasure" and "satisfaction"), or exhibits sentiment recognition bias on image and text samples containing complex visual cues.

[0007] Therefore, there is an urgent need for a method that can effectively alleviate advantage collapse and improve the efficiency and stability of verifiable reward reinforcement learning in emotion classification tasks, so as to achieve higher accuracy and more stable multimodal emotion recognition results. Summary of the Invention

[0008] The first objective of this invention is to address the existing problems and shortcomings by proposing an emotion classification method based on visual language models and conditional reasoning. This method enhances the training efficiency and stability of verifiable reward reinforcement learning, overcomes the "dominance collapse" problem in verifiable reinforcement learning, and strengthens the emotion classification ability of visual language models.

[0009] To address the aforementioned problems, the technical solution adopted in this invention is to guide the visual language model to output different answers, thereby avoiding ineffective training. Specifically:

[0010] Obtain text-image pairs and label them with sentiment tags to form a sentiment tag set S;

[0011] Using a visual language model as the policy model, multiple general inferences are performed on the text-image pair to generate a set of general inference representations X and corresponding sentiment prediction labels Y; at the same time, conditional inferences are performed on each sentiment label in the sentiment label set to generate a conditional inference result R between the text-image pair and the sentiment label.

[0012] The general reasoning representation X is combined with the sentiment prediction label Y, and the conditional reasoning result R is combined with the sentiment label set S to form the response sample D;

[0013] The reward value and advantage estimate are calculated based on the response sample D. The policy model is then optimized using verifiable reward reinforcement learning to obtain the optimized policy model.

[0014] The optimized strategy model is used to predict the sentiment of new text-image pairs, and the final sentiment classification result is output.

[0015] Furthermore, in the conditional reasoning process, for each input text-image pair And each sentiment tag in the sentiment tag set S , to image ,text And this sentiment tag Together, they form a structured prompt for conditional reasoning to determine the relevance of the text image to the given context. With hypothetical sentiment labels The relationship between them is adopted The model represents text-image pairs and sentiment labels. The reasoning results regarding the relationship between them:

[0016]

[0017] The inference results corresponding to all candidate labels are combined into a set. .

[0018] Furthermore, the response sample D is represented as:

[0019]

[0020] Where x' represents the result of reasoning and y' represents the label; the label includes two parts, one is the label predicted by the model, and the other is the label of the model constructed during the conditional reasoning process, which is consistent with the label predicted by the model in form.

[0021] Secondly, the present invention provides an emotion classification system for implementing the above method, comprising the following modules:

[0022] The data processing module retrieves the text-image pairs to be categorized.

[0023] The sentiment inference module uses an optimized strategy model to predict the sentiment of text-image pairs and outputs the sentiment classification results.

[0024] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the method.

[0025] Fourthly, the present invention provides a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the method.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] This invention introduces conditional reasoning into the traditional visual language model reasoning, giving the model's thinking process a clear "hypothesis-verification" structure. That is, in each step of the reasoning, the model focuses on whether the text image expresses emotion. This assumption leads to further development. This explicit assumption-driven approach not only makes the model's internal logic more aligned with human cognition, but also ensures that each reasoning outcome... All of these can be directly mapped to the corresponding sentiment hypothesis, thus forming an interpretable inference chain. Furthermore, conditional inference, through label-by-label reasoning and comparison, enables the model to perform fine-grained alignment at the semantic level, evaluating the degree of matching between "image sentiment," "text sentiment," and "hypothetical sentiment" in both visual and linguistic modalities. This label-by-label comparative inference strengthens the model's ability to distinguish between similar sentiment categories (e.g., "pleasure" and "satisfaction," "irony" and "humor"), thereby improving overall classification accuracy. This overcomes the shortcomings of traditional multi-classification models, which often directly predict result labels in the implicit sentiment space, lack explicit comparison of candidate sentiments, and are easily interfered with by noisy text or complex visual content.

[0028] Furthermore, this invention reorganizes the general reasoning process and conditional reasoning process of the strategy model into a set of response samples, which ensures that each set of responses contains different classification labels, thus solving the problem of advantage collapse. Attached Figure Description

[0029] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0031] As mentioned earlier, existing research has sufficiently demonstrated that excessively high policy gradient variance is one of the key factors leading to instability or even collapse in reinforcement learning training. The sparsity of reward signals further amplifies this problem: when high-quality feedback signals only appear in a very small number of samples or during the inference phase, the gradient estimates relied upon by the model during policy updates will exhibit significant volatility. Furthermore, the differences between general reasoning and conditional reasoning in terms of thought chain structure, inference depth, and path dependence can also lead to inconsistencies in gradient propagation paths, thus exacerbating the variance accumulation problem in policy optimization.

[0032] To avoid high gradient variance caused by inconsistencies between conditional inference content and general inference content, this invention constrains the inference content, specifically:

[0033] First, the general reasoning process is divided into three stages: text extraction and perception, multimodal semantic understanding, and cross-modal reasoning and decision-making.

[0034] Specifically, the model first identifies and extracts textual information from images to perceive the main content of the images; then, it analyzes and models the correlation between images and text at the semantic level; finally, it generates an answer or judgment output based on the reasoning results. This phased design helps to explicitly decompose the thought process, thereby constructing a cognitive path similar in structure to the general reasoning process during conditional reasoning, effectively mitigating the instability of policy gradients and improving the convergence of reinforcement learning optimization.

[0035] To address the sparsity issue of reward signals, a hierarchical reward design mechanism is introduced based on the aforementioned phased inference process. Specifically, intermediate reward signal points are set at key stages such as perception, cognitive reasoning, and decision-making, enabling the model to receive targeted optimization feedback at each sub-stage. This design not only provides denser supervision signals in long inference chains, thus alleviating the gradient vanishing and excessive variance problems caused by sparse terminal rewards in traditional reinforcement learning, but also guides the model to gradually form a stable inference path from low-level perception to high-level decision-making during training. Ultimately, the phased reward mechanism improves the convergence speed and stability of the model in multimodal inference tasks while ensuring overall task-oriented consistency.

[0036] The present invention will be further described below.

[0037] like Figure 1 As shown, this invention provides a sentiment classification method based on a visual language model, comprising the following steps:

[0038] S1, retrieve text-image pairs and sentiment tags; define a set of sentiment category tags. (e.g., positive, negative, or neutral), the labels for all subsequent samples are in the set. In Chinese, retrieve text-image pairs. The text and image pairs were then labeled with sentiment tags. .

[0039] Text-image pairs typically utilize publicly available, high-quality multimodal sentiment classification datasets. For stable training, the original images may suffer from significant resolution variations, inconsistent aspect ratios, or background noise. To avoid excessively large input sizes leading to memory overflow or excessively small sizes causing information loss, the images are uniformly resized to a specified size while maintaining consistent aspect ratios to preserve key visual semantic structures and obtain the desired image input. Text input These texts typically originate from social media posts, captions, or comments, and may contain emojis, colloquial expressions, or non-standard phrases. To improve the model's understanding and semantic stability, the text needs to be normalized: HTML tags, URLs, special characters, and repeated punctuation should be removed; capitalization and space formatting should be standardized; key sentiment words and modifiers should be retained to maintain emotional signals.

[0040] S2 utilizes generative visual language models (such as Qwen2.5-VL and InternVL3) as policy models in multimodal sentiment analysis tasks. Initialize the strategy model and process the collected text and image data. Perform joint reasoning and sentiment prediction. (Strategy Model) Based on image input With text input Perform n inferences to generate a set of general inference representations related to sentiment judgments. :

[0041]

[0042] in This represents the multimodal semantic and emotional interaction representations formed during the reasoning process of a generative visual language model.

[0043] Subsequently, the model is represented by this general reasoning. Output sentiment prediction tags: ,in This is a set of sentiment labels (such as positive, negative, or neutral). Therefore, for each input text-image pair... Visual language models can all yield a set of general reasoning representations. and the corresponding sentiment prediction results ,Right now:

[0044]

[0045] To fully unleash the model's cross-modal reasoning capabilities, it is necessary to design structured and heuristic prompts to guide the model to focus on the task objective during the output phase.

[0046] Specifically, the prompt template constructed in this embodiment includes three parts:

[0047] Context description: Provide the model with the scene background or semantic theme of the image and text, such as "Given an image and its caption, please determine the overall sentiment".

[0048] Task instructions: Explicitly specify the model's inference objective. To ensure controllable output content, the classification objective needs to be explicitly written in the prompt, such as "Analyze the semantic relationship between the image and text, and determine the emotion category (positive / negative / sarcastic, etc.)".

[0049] Output constraints: Ensure the model generates consistent, parsable, and structured outputs of reasons and sentiment categories. Typical output formats for visual language models include: <reason>Inference process< / reason> <answer>Answer< / answer> The reasoning process corresponds to The answer corresponds to Through the aforementioned prompt design, the visual language model can not only understand the multimodal input content but also perform high-level semantic reasoning (such as sentiment consistency, metaphor detection, and visual irony recognition) in a unified semantic space, generating output results that include explicit predicted labels and implicit reasoning processes. To accelerate the reasoning process, frameworks such as VLLM are typically used to speed up the inference process.

[0050] S3, conditional reasoning:

[0051] For each input sample To determine the degree of association between input samples and different sentiment hypotheses, the model needs to perform conditional inference during the inference phase. Specifically, for a set of sentiment labels... Each candidate sentiment tag The image, text, and sentiment tag are combined to construct a structured prompt for conditional reasoning, such as: "Given an image and its text, assume that the content expresses the sentiment..." Please analyze the reasons to determine the impact of the text image. With hypothetical sentiment labels The visual language model then performs inference based on the possible relationships between the input conditions, generating the corresponding inference results. , represented as:

[0052]

[0053] in Includes the model's sentiment label for this hypothesis. Explain the conclusion and reasoning process. Then, explain the reasoning result. and their corresponding emotional tags Constructing conditional inference samples .

[0054] The inference results corresponding to all candidate labels are combined into a set. ,in Representing the visual language model in the sentiment label space The overall set of reasoning results. The introduction of conditional reasoning has significant advantages in multimodal sentiment understanding tasks. This is achieved by explicitly introducing candidate sentiment labels into the input. The strategy model can focus on the potential association between text-image pairs and specific emotional semantics in a goal-oriented manner during the inference stage, thereby significantly improving the model's discriminative ability and inference controllability.

[0055] S4, compares the samples from S2 and S3. Reasoning samples and conditional reasoning samples Combined into a set of reasoning samples :

[0056]

[0057] By constructing inference samples using two different inference methods, the model ensures that there are different answers within the group for both simple and difficult samples. This avoids the model's over-reliance on a single inference path or answer template during training, improving its robustness and generalization ability under diverse inference strategies. Specifically, this design forces the model to focus on the differences in the inference logic itself rather than simply memorizing conclusions, effectively mitigating the tendency for answer-driven learning and promoting a process-driven inference mechanism. Simultaneously, the differences in answers within the group provide a natural contrast signal for the reinforcement learning phase, enabling the reward function to more accurately distinguish between high-quality and low-quality inference paths, further stabilizing the training process. These results characterize the model's understanding and inference distribution of input samples from different emotional perspectives, providing diverse and verifiable outputs for subsequent reward calculation and advantage estimation.

[0058] S5 is a typical verifiable reward reinforcement learning process, which includes the following steps:

[0059] Follow the general steps of verifiable reward reinforcement learning to develop a policy model. To optimize, the first step is to calculate based on rules. Reward value for each sample In classification tasks, a reward of 1 is given if the prediction is correct, and a reward of 0 is given otherwise.

[0060]

[0061] express The true label, according to S4, regardless of What is the true value? In each group of response samples, one sample will receive a reward value of 1, while the other samples will receive a reward value of 0.

[0062] Next, the standardized reward is calculated based on the reward variance within the sample group:

[0063]

[0064]

[0065]

[0066] Based on the above, each set of reward values Certainly not equal to In this way, regardless of whether it is a simple or difficult sample, each sample in each group of response samples will obtain a different advantage estimate. ( ).

[0067] Based on dominance value The policy parameters are updated using the Group Relative Policy Optimization (GRPO) algorithm; this verifies the optimization objective of reward reinforcement learning. Standard formula:

[0068] First, define the ratio. :

[0069]

[0070] Then the optimization objective

[0071]

[0072] yes The set of parameters, This is the clipping ratio, used for stable training. According to formula (10), the optimization objective is used to update the policy model parameters through backpropagation. . because The value of is not 0, so the policy model will obtain a valid gradient with each update.

[0073] S6: Repeat S2-S5 until convergence or the preset number of training rounds is reached, and then use the optimized model to predict sentiment for new text-image pairs.

[0074] To verify the effectiveness and generality of the proposed method, experiments were conducted on different types of multimodal pedestal models, including the representative visual language models Qwen2.5-VL and LLaVA 1.5. These models differ significantly in architecture, visual encoding methods, and model pre-training. Therefore, experiments on different pedestals can comprehensively evaluate the performance of the proposed method in terms of multimodal semantic alignment, inference stability, and cross-modal generalization ability.

[0075] In the comparative experiments, the MMSD dataset was used. MMSD is a publicly available dataset for researching multimodal sarcasm detection tasks. First proposed by Hasan et al. in 2021, this dataset aims to advance research on the automatic understanding and modeling of text-image sarcasm in social media. The MMSD dataset is primarily collected from the Twitter platform and contains multimodal samples with both text (tweet content) and images (accompanying images). Researchers ensured that the sarcasm semantics were clear and the cross-modal relationships were rich through multiple rounds of manual annotation and filtering. Each sample consists of three parts: 1. Text Modality: The text content of the tweet, which may contain emojis, colloquial expressions, irony, or exaggeration; 2. Image Modality: The image accompanying the tweet, which may be used to reinforce or reverse the emotional semantics of the text; 3. Label: A binary label, 1 for sarcastic and 0 for non-sarcastic.

[0076] The experimental results are shown in Table 1.

[0077] Table 1

[0078] Model Acc. (%) P (%) R (%) F1 (%) Text-Only Methods TextCNN (Kim, 2014) 71.61 64.62 75.22 69.52 BiLSTM (Zhou et al., 2016) 72.48 68.02 68.08 68.05 SMSD (Xiong et al., 2019) 73.56 68.45 71.55 69.97 RoBERTa (Liu et al., 2019) 79.66 76.74 75.70 76.21 ChatGLM2-6B (Du et al., 2022) 78.41 78.15 78.65 78.23 ChatGLM2-6B (Du et al., 2022) 80.08 80.52 81.04 80.04 LLaMA2-7B (Touvron et al., 2023b) 82.52 82.15 82.46 82.27 LLaMA2-7B (Touvron et al., 2023b) 84.68 84.40 84.94 84.53 ' Image-Only Methods 65.50 61.17 54.39 57.58 ResNet (He et al., 2016) 72.02 65.26 74.83 69.72 ViT (Dosovitskiy et al., 2020) 70.57 64.84 69.05 66.88 HFM (Cai et al., 2019) 80.03 76.28 77.82 77.04 Att-BERT (Pan et al., 2020) 79.83 75.82 78.01 76.90 CMGCN (Liang et al., 2022) 76.50 73.48 71.07 72.25 HKE (Liu et al., 2022a) 71.40 71.80 72.17 71.34 DynRT-Net (Tian et al., 2023) 85.64 80.33 88.24 84.10 Multi-view CLIP (Qin et al., 2023) 85.18 85.89 85.20 85.11 LLaVA1.5-7B (Liu et al., 2023a) 86.43 87.00 86.30 86.34 LLaVA1.5-7B GRPO 81.23 72.98 89.58 80.43 Qwen2.5VL GRPO 87.23 86.56 88.34 87.44 LLaVA1.5-7B Ours Qwen2.5VL Ours 82.60 74.33 91.03 81.83

[0079] The results demonstrate the effectiveness of the method of the present invention on different substrates. Compared with the original GRPO, it improves by 1.37 on the Qwen2.5VL7B substrate and by 0.8 on the LLaVA1.5 7B substrate.

[0080] The above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A method for sentiment classification based on visual language model and conditional reasoning, characterized in that, The emotion classification method comprises the following steps: Obtain a text picture pair, and label emotions of the text picture pair to form an emotion label set S; Use a visual language model as a strategy model to perform multiple general inferences on the text picture pair to generate a set of inference representations X and corresponding emotion prediction labels Y; meanwhile, perform conditional inference on each emotion label in the emotion label set to generate conditional inference results R between the text picture pair and the emotion label; Combine the inference representations X and the emotion prediction labels Y, and the conditional inference results R and the emotion label set S to form a response sample D; Calculate a reward value and an advantage estimation value based on the response sample D, optimize the strategy model by using a verifiable reward reinforcement learning, and obtain an optimized strategy model; Perform emotion prediction on a new text picture pair by using the optimized strategy model, and output a final emotion classification result.

2. The emotion classification method of claim 1, wherein, In the conditional inference process, for each input text picture pair and each sentiment label in the sentiment label set S , the image , the text and the sentiment label are jointly constructed into a structured prompt for conditional inference to determine the association between the text picture pair and the hypothetical sentiment label ; the inference result of the relationship between the text picture pair and the sentiment label is represented by a model . combining the inference results corresponding to all candidate labels into a set .

3. The sentiment classification method of claim 1, wherein, The response sample D is expressed as: Wherein x' represents an inference result, and y' represents a label.

4. The sentiment classification method of claim 1, wherein, The optimization of the strategy model comprises the following steps: First, the reward value of each sample in the response sample D is calculated In the classification task, if the prediction is correct, the reward value is 1, otherwise the reward value is 0; Next, the normalized reward is calculated based on the reward variance, and the advantage estimation value of the response sample D is obtained ; Based on the advantage estimate The strategy parameters are updated using a group relative policy optimization algorithm.

5. The sentiment classification method of claim 4, wherein, The updating of the strategy parameters is specifically: Verifiable reward reinforcement learning to optimize objectives Standard formula: First define the ratio : Then the optimization objective wherein is a parameter set of is a clipping ratio for stabilizing training; the optimization objective is updated by backpropagation to update the policy model parameters according to equation (10).

6. An affective classification system implementing the method according to any one of claims 1 to 5, characterized in that, Comprise the following modules: A data processing module that obtains a text picture pair to be classified; An emotion inference module that performs emotion prediction on the text picture pair by using the optimized strategy model, and outputs an emotion classification result.

7. An electronic device comprising a processor and a memory, characterized in that The memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method according to any one of claims 1-5.

8. A machine-readable storage medium, characterized in that, The machine readable storage medium stores machine executable instructions, and when the machine executable instructions are called and executed by the processor, the machine executable instructions cause the processor to implement the method according to any one of claims 1-5.

Images