Multi-modal emotion recognition model training method and device, equipment and medium

Abstract

The invention provides a multi-modal emotion recognition model training method and device, equipment and a medium. The method comprises the following steps: initializing a pre-trained multi-modal large language model to obtain a reference model and a to-be-trained strategy model; inputting the multi-modal emotion recognition sample into a to-be-trained strategy model to obtain a prediction text, and inputting the multi-modal emotion recognition sample into a reference model to obtain a reference text; obtaining the loss corresponding to the prediction text according to the difference between the prediction text and the reference text; obtaining an interpretation perception reward based on an emotion interpretation label of the multi-modal emotion recognition sample, and obtaining an emotion category accuracy reward based on an emotion category label of the multi-modal emotion recognition sample and a prediction emotion category in the prediction text; fusing the interpretation perception reward and the emotion category accuracy reward to obtain a total reward; and performing parameter updating on the strategy model based on the total reward and the loss corresponding to the prediction text to obtain a multi-modal emotion recognition model.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and sentiment analysis technology, specifically to a training method, apparatus, device, and medium for a multimodal sentiment recognition model. Background Technology

[0002] Multimodal Emotion Recognition (MER) aims to infer human emotional states by fusing information from multiple modalities such as text, video, and audio. It is one of the core technologies in fields such as human-computer interaction and mental health monitoring. With the development of Multimodal Large Language Models (MLLMs), Explainable Multimodal Emotion Recognition (EMER) has become a research hotspot. Its goal is not only to accurately predict emotion categories, but also to require the model to generate interpretable feedback about the causes of emotions, thereby enhancing the model's transparency and user trust.

[0003] Related methods (such as Emotion-LLaMA and AffectGPT) improve MER performance through supervised fine-tuning or the introduction of reinforcement learning. However, these methods still have significant drawbacks in generating sentiment interpretations: on the one hand, the supervised fine-tuning paradigm is prone to causing the model to produce illusions or logical inconsistencies when generating inference content; on the other hand, the related reinforcement learning methods mainly focus on optimizing the accuracy of sentiment recognition, and sentiment interpretation is usually only implicitly processed as a byproduct of optimization, lacking an effective supervision mechanism for interpretation quality, resulting in interpretations that are not accurate or clear enough, and are difficult to meet the interpretability requirements of practical applications. Summary of the Invention

[0004] The purpose of this application is to provide a training method, apparatus, device, and medium for a multimodal emotion recognition model, which can solve the problem that the lack of effective quality supervision in the generation of emotion interpretation in the prior art leads to the inaccurate and unclear emotion interpretation generated by the model.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: A first aspect of this application discloses a method for training a multimodal emotion recognition model, the method comprising: Obtain multimodal emotion recognition samples, wherein the modalities of the multimodal emotion recognition samples include at least two of the following modalities: text modality, video modality, and audio modality; The pre-trained multimodal large language model is initialized to obtain a reference model and a policy model to be trained; The multimodal emotion recognition samples are input into the policy model to be trained to obtain predicted text, and the multimodal emotion recognition samples are input into the reference model to obtain reference text; The loss corresponding to the predicted text is obtained based on the difference between the predicted text and the reference text; Based on the sentiment interpretation labels of the multimodal sentiment recognition samples, the interpretation perception reward corresponding to the predicted text is obtained; and based on the sentiment category labels of the multimodal sentiment recognition samples and the predicted sentiment category in the predicted text, the sentiment category accuracy reward corresponding to the predicted text is obtained. The total reward for the predicted text is obtained by fusing the interpretation perception reward and the sentiment category accuracy reward corresponding to the predicted text. Based on the total reward and loss corresponding to the predicted text, the parameters of the policy model to be trained are updated to obtain a multimodal emotion recognition model.

[0006] Optionally, based on the sentiment interpretation labels of the multimodal sentiment recognition samples, the interpretation-aware reward of the predicted text is obtained, including: Using each ROUGE index in the ROUGE index set, determine the sentiment interpretation label of the multimodal sentiment recognition sample and the score of the predicted sentiment interpretation on that ROUGE index; The average score of the sentiment interpretation label of the multimodal sentiment recognition sample and the predicted sentiment interpretation on the ROUGE index set is used as the interpretation perception reward of the predicted text. Based on the sentiment category labels of the multimodal sentiment recognition samples and the predicted sentiment category in the predicted text, a sentiment category accuracy reward for the predicted text is obtained, including: The predicted sentiment category in the predicted text is checked to see if it matches the sentiment category label of the multimodal sentiment recognition sample, and the sentiment category accuracy reward of the predicted text is obtained.

[0007] Optionally, based on the sentiment interpretation labels of the multimodal sentiment recognition samples, the interpretation-aware reward of the predicted text is obtained, including: Using each ROUGE index in the ROUGE index set, determine the sentiment interpretation label of the multimodal sentiment recognition sample and the score of the predicted sentiment interpretation on that ROUGE index; The average score of the sentiment interpretation label of the multimodal sentiment recognition sample and the predicted sentiment interpretation on the ROUGE index set is used as the interpretation perception reward of the predicted text. Based on the sentiment category labels of the multimodal sentiment recognition samples and the predicted sentiment category in the predicted text, a sentiment category accuracy reward for the predicted text is obtained, including: The predicted sentiment category in the predicted text and the sentiment category label of the multimodal sentiment recognition sample are respectively mapped to sectors of the sentiment wheel; Detect whether there is semantic overlap between the sector to which the predicted sentiment category is mapped and the sector to which the sentiment category label is mapped, in order to obtain the sentiment category accuracy reward of the predicted text.

[0008] Optionally, the method further includes: The quality of the predicted sentiment interpretation is evaluated using the reference model through the following steps to obtain the perceived reward of the predicted text: Evaluate the correlation between the predicted sentiment interpretation in the predicted text and the predicted sentiment category in the predicted text; Evaluate the logical consistency among the different modalities of predicted sentiment interpretation cues included in the predicted sentiment interpretation; Evaluate the contribution of the predicted sentiment interpretation cue of each modality in the predicted sentiment interpretation to the predicted sentiment category in the predicted text.

[0009] Optionally, the predicted text and the reference text each include T tokens, where T is an integer greater than 1; the method further includes: The t-th word in the predicted text is evaluated from 1 to T in sequence, based on three dimensions: pleasantness, arousal, and dominance, to determine the sentiment polarity score of that word. The loss corresponding to the predicted text is obtained based on the difference between the predicted text and the reference text, including: By taking t sequentially from 1 to T, the difference measure of the t-th word in the predicted text is obtained based on the difference between the t-th word in the predicted text and the t-th word in the reference text. The weighted weight of the t-th word in the predicted text is obtained by using the sentiment polarity score of the t-th word in the predicted text as the weight and combining it with the difference measure of the T words in the predicted text. Based on the weighted weight of the t-th word in the predicted text, the loss of the t-th word is weighted to obtain the loss corresponding to the predicted text.

[0010] Optionally, the t-th lexical unit in the predicted text is evaluated from three dimensions: pleasantness, arousal, and dominance, to determine the sentiment polarity score of that lexical unit, including: For each word element in the predicted text, the pleasantness, arousal, and dominance of that word element are determined using a pleasantness-arousal-dominance dictionary. The maximum value among the pleasantness, arousal and dominance of the word element is determined as the dominant emotional polarity score of the word element. The overall emotional polarity score of the word element is obtained by combining the pleasantness, arousal and dominance of the word element. The emotional polarity score of a word is obtained based on its dominant emotional polarity score and its overall emotional polarity score.

[0011] A second aspect of this application discloses a training apparatus for a multimodal emotion recognition model, the apparatus comprising: The sample acquisition module is used to obtain multimodal emotion recognition samples, wherein the modalities of the multimodal emotion recognition samples include at least two of the following modalities: text modality, video modality, and audio modality; The initialization module is used to initialize the pre-trained multimodal large language model to obtain the reference model and the policy model to be trained. The sample input module is used to input the multimodal emotion recognition sample into the policy model to be trained to obtain predicted text, and to input the multimodal emotion recognition sample into the reference model to obtain reference text; The loss calculation module is used to obtain the loss corresponding to the predicted text based on the difference between the predicted text and the reference text. The reward calculation module is used to obtain the interpretation perception reward corresponding to the predicted text based on the sentiment interpretation label of the multimodal sentiment recognition sample, and to obtain the sentiment category accuracy reward corresponding to the predicted text based on the sentiment category label of the multimodal sentiment recognition sample and the predicted sentiment category in the predicted text. The reward fusion module is used to fuse the interpretation perception reward and the sentiment category accuracy reward corresponding to the predicted text to obtain the total reward corresponding to the predicted text. The parameter update module is used to update the parameters of the policy model to be trained based on the total reward and the loss corresponding to the predicted text, so as to obtain a multimodal sentiment recognition model.

[0012] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the training method for the multimodal emotion recognition model described in the first aspect of this application.

[0013] A fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for the multimodal emotion recognition model described in the first aspect of this application.

[0014] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the training method for the multimodal emotion recognition model described in the first aspect of this application.

[0015] The embodiments of this application have the following advantages: In this embodiment, by introducing an explanation-aware reward and an emotion category accuracy reward, and fusing them into a total reward, combined with the difference loss between the policy model and the reference model, the model is jointly optimized within a reinforcement learning framework. This effectively improves the performance of the multimodal emotion recognition model in terms of both emotion recognition accuracy and emotion interpretation quality. Specifically, the explanation-aware reward supervises the emotion interpretations generated by the model based on emotion interpretation labels, guiding the model to generate more accurate, clear, and high-quality interpretations relevant to the causes of emotions. The emotion category accuracy reward constrains the predicted emotion category based on emotion category labels, ensuring the accuracy of emotion recognition. The fusion of these two rewards allows the model to balance recognizability and interpretability during the optimization process, avoiding the problem of insufficient interpretation quality caused by emotion interpretation being implicitly optimized as a byproduct in related methods. Simultaneously, combining the difference loss between the reference model and the policy model further stabilizes the training process, resulting in an emotion recognition model that outputs both accurate and interpretable emotion analysis results, thereby enhancing the model's transparency and user trust. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the steps of a training method for a multimodal emotion recognition model provided in an embodiment of this application. Figure 2 This is an overall architecture diagram of a training method for a multimodal emotion recognition model provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the steps of an emotion polarity weighting scheme provided in an embodiment of this application; Figure 4This is a flowchart illustrating the steps of a self-assessment mechanism scheme provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a training device for a multimodal emotion recognition model provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The training method, apparatus, device, and medium of the multimodal emotion recognition model provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0020] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a training method for a multimodal emotion recognition model provided in this application embodiment. The training method for a multimodal emotion recognition model provided in this application embodiment may include steps S110 to S170: Step S110: Obtain multimodal emotion recognition samples, wherein the modalities of the multimodal emotion recognition samples include at least two of the following modalities: text modality, video modality, and audio modality.

[0021] In this step, the multimodal emotion recognition sample is raw data containing information from multiple modalities, specifically a combination of at least two modalities: text, video, and audio. For example, a multimodal emotion recognition sample may simultaneously contain a dialogue text (text modality), a video clip of the speaker's face (video modality), and the corresponding speech signal (audio modality). These samples are typically derived from publicly available emotion datasets (e.g., IEMOCAP, CMU-MOSI). In addition to containing the original multimodal content, each multimodal emotion recognition sample is pre-labeled with emotion category tags (e.g., "happy," "sad") and detailed emotion explanation tags (e.g., textual descriptions of the reasons for the emotion, facial expressions, and tone of voice) for subsequent supervised training.

[0022] Step S120: Initialize the pre-trained multimodal large language model to obtain the reference model and the policy model to be trained.

[0023] In this step, the model is trained using a reinforcement learning framework. Specifically, a pre-trained multimodal large language model, such as Qwen2.5-Omni-7B, is selected as the base model. Two copies of this pre-trained multimodal large language model are made, one initialized as a reference model and the other as the policy model to be trained. The policy model is the target model whose parameters are updated during subsequent training, with the goal of generating high-quality sentiment predictions and interpretations. The reference model maintains relatively fixed parameters during training to provide a stable benchmark for updating the policy model, preventing drastic fluctuations in model training.

[0024] Step S130: Input the multimodal emotion recognition sample into the policy model to be trained to obtain the predicted text, and input the multimodal emotion recognition sample into the reference model to obtain the reference text.

[0025] In this step, the same multimodal sentiment recognition sample is input into both the policy model and the reference model. Both models process the input multimodal information, including text, video, and audio, and output corresponding predicted text. This predicted text is typically organized in a preset format, such as including explanatory text for the sentiment reasoning process. <think>Part and used to output the final sentiment category prediction <answer>part.

[0026] It should be noted that the reference text output by the reference model and the predicted text output by the policy model maintain the same format, but the content may differ due to variations in model parameters. The reference text is used to subsequently measure the degree of deviation from the policy model's output.

[0027] Step S140: Based on the difference between the predicted text and the reference text, obtain the loss corresponding to the predicted text.

[0028] In this step, to measure whether the output of the policy model has deviated excessively, the difference between the predicted text and the reference text is calculated. Since the reference model is relatively stable, its output can serve as a constraint on the policy model's update. By comparing the differences in the output probability distribution or semantic representation of the two at the same position (such as a single word), the loss corresponding to the policy model's output can be calculated. The role of this loss term is to constrain the update magnitude of the policy model during reinforcement learning optimization, preventing it from causing semantic collapse or generating uncontrollable content due to excessive pursuit of high rewards, thereby ensuring the stability of the training process.

[0029] Step S150: Based on the sentiment interpretation label of the multimodal sentiment recognition sample, obtain the interpretation perception reward corresponding to the predicted text; and based on the sentiment category label of the multimodal sentiment recognition sample and the predicted sentiment category in the predicted text, obtain the sentiment category accuracy reward corresponding to the predicted text.

[0030] In this step, in order to guide the model to generate high-quality sentiment analysis results, two types of reward signals were designed: "explanation perception reward" and "sentiment category accuracy reward".

[0031] The perceived reward is used to evaluate the quality of the sentiment interpretation generated by the policy model. The sentiment interpretation portion (e.g., ...) is extracted from the predicted text. <think>The model generates an explanation (containing the content of the label) and compares it with the pre-labeled sentiment interpretation labels of multimodal sentiment recognition samples. By measuring the semantic consistency between the two, the accuracy and clarity of the explanation generated by the model in reflecting the true causal clues of sentiment can be quantified.

[0032] The sentiment category accuracy reward is used to evaluate the accuracy of the policy model on the sentiment recognition task. (This involves extracting information from the predicted text.) <answer>The model identifies the predicted sentiment category within a sample and compares it with pre-labeled sentiment category information from multimodal sentiment recognition samples. By determining whether the predicted category is correct, the model receives a corresponding reward signal.

[0033] It should be noted that there are multiple ways to calculate the above two types of rewards, such as calculation based on semantic similarity or calculation based on hierarchical sentiment mapping. This embodiment does not limit this, and subsequent embodiments will provide a variety of optional implementation methods.

[0034] Step S160: The interpretation perception reward and sentiment category accuracy reward corresponding to the predicted text are fused to obtain the total reward corresponding to the predicted text.

[0035] In this step, the interpretation perception reward and the sentiment category accuracy reward calculated in the previous steps are fused to obtain a comprehensive total reward. The fusion method can be weighted summation, product, or other operations that can comprehensively reflect the importance of the two rewards. This total reward serves as an optimization direction within the reinforcement learning framework, taking into account both the accuracy of sentiment recognition and the quality of sentiment interpretation. This ensures that the model, during training, does not simply pursue correct classification, but rather strives to "say it right" and "say it well."

[0036] Step S170: Based on the total reward corresponding to the predicted text and the loss corresponding to the predicted text, update the parameters of the policy model to be trained to obtain a multimodal emotion recognition model.

[0037] In this step, the parameters of the policy model are updated by combining the loss calculated in step S140 and the total reward calculated in step S160. Specifically, in reinforcement learning frameworks (such as Group Relative Policy Optimization, GRPO), the total reward is used to calculate the advantage value of each generated sample, while the loss corresponding to the predicted text is used to constrain the update magnitude.

[0038] Through multiple iterations of the above steps, the policy model, guided by reward signals, gradually learns to generate more accurate sentiment categories and higher-quality sentiment interpretations, while avoiding catastrophic forgetting or semantic collapse through loss constraints. After training, a parameter-optimized multimodal sentiment recognition model is obtained. This model can receive multimodal inputs and output accurate and interpretable sentiment analysis results.

[0039] The technical solution adopted in this embodiment introduces both interpretation perception rewards and sentiment category accuracy rewards, incorporating the accuracy of sentiment recognition and the quality of sentiment interpretation into the optimization objective. This allows the model to focus not only on the correctness of sentiment classification during training but also on the accuracy and clarity of the generated sentiment interpretation. Compared to related methods that only optimize recognition accuracy, the model trained in this embodiment can output more persuasive and transparent sentiment analysis results, effectively improving the model's interpretability and user trust.

[0040] In some embodiments, a standard format reward is also included to enforce the required Chain-of-Thought (CoT) structure. <think>< / think> <answer>< / answer> If the model's output format is correct, a larger reward value (e.g., 1) is assigned; if the output format is incorrect, a smaller reward value (e.g., 0) is assigned. For example, a standard format reward... It can be represented as:

[0041] Thus, the standard format reward, the interpretation-aware reward corresponding to the predicted text, and the sentiment category accuracy reward are fused to obtain the total reward corresponding to the predicted text. Based on the total reward corresponding to the predicted text and the loss corresponding to the predicted text, the parameters of the multimodal large language model to be trained are updated to obtain the multimodal sentiment recognition model. This ensures that each policy update is in a direction that is both structurally sound and conducive to subsequent interpretation.

[0042] In one optional embodiment, the specific calculation methods for the two types of rewards involved in step S150 above are described in detail. This embodiment provides a calculation method for interpretation perception rewards based on emotion interpretation labels, and a calculation method for accuracy rewards based on emotion category matching.

[0043] The step S150, "obtaining the interpretation-aware reward of the predicted text based on the sentiment interpretation labels of the multimodal sentiment recognition samples," may include the following steps A1 to A2: Step A1: Using each ROUGE index in the ROUGE index set, determine the sentiment interpretation label of the multimodal sentiment recognition sample and the score of the predicted sentiment interpretation on that ROUGE index.

[0044] ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics commonly used to evaluate the quality of automatically generated text, primarily measuring the similarity between the generated text and the labeled text. In this embodiment, the ROUGE metric set is used to quantitatively evaluate the sentiment interpretation generated by the model. This metric set specifically includes several metrics such as ROUGE-1 (based on unary word overlap), ROUGE-2 (based on bigram word overlap), and ROUGE-L (based on longest common subsequence).

[0045] In practical implementation, firstly, pre-labeled sentiment explanation tags are extracted from multimodal sentiment recognition samples. These tags are textual descriptions of the cause of the sample's sentiment, such as "the speaker's lips are upturned, their tone is light, and they express a pleasant emotion." Simultaneously, the predicted sentiment explanation portion is extracted from the predicted text output by the policy model, i.e., the part located in... <think>The inference content within the label is then analyzed. Subsequently, a similarity score between the sentiment interpretation label and the predicted sentiment interpretation is calculated using each ROUGE metric from the ROUGE metric set. The output value of each ROUGE metric is between 0 and 1, with higher values ​​indicating a greater degree of overlap between the predicted sentiment interpretation and the sentiment interpretation label on that metric dimension.

[0046] Step A2: The average score of the sentiment interpretation label of the multimodal sentiment recognition sample and the predicted sentiment interpretation on the ROUGE index set is used as the interpretation perception reward of the predicted text; After obtaining the scores for each ROUGE metric, the arithmetic mean of these scores is calculated as the interpretive reward for the predicted text. For example, the interpretive reward for the predicted text... The calculation formula can be expressed as:

[0047] in, This indicates the set of ROUGE metrics used. , and These represent the scores for the three metrics: ROUGE-1, ROUGE-2, and ROUGE-L. For example, if the ROUGE-1 score is 0.65, the ROUGE-2 score is 0.52, and the ROUGE-L score is 0.60, then the perceptual reward for the predicted text is (0.65 + 0.52 + 0.60) / 3 = 0.59. This reward value reflects the overall degree of agreement between the sentiment interpretation generated by the model and the real sentiment interpretation at the lexical, phrase, and sentence structure levels.

[0048] It should be noted that the selection and number of ROUGE metrics can be adjusted according to actual needs. For example, only the ROUGE-L metric can be used, or other text evaluation metrics such as BLEU and METEOR can be added. This embodiment does not limit this. Furthermore, the above-described method for calculating the perceived reward is a label-based supervised implementation. In practical applications, if there is a lack of sentiment interpretation labels, a self-evaluation method can also be used for reward calculation. The specific implementation method will be described in detail in subsequent embodiments (steps E1 to E3).

[0049] The step S150, "obtaining the sentiment category accuracy reward of the predicted text based on the sentiment category label of the multimodal sentiment recognition sample and the predicted sentiment category in the predicted text," may include the following step B1: Step B1: Detect whether the predicted sentiment category in the predicted text can match the sentiment category label of the multimodal sentiment recognition sample, and obtain the sentiment category accuracy reward of the predicted text.

[0050] In practical implementation, the predicted sentiment category is first extracted from the predicted text output by the policy model. This category is located in... <answer>Within the tags, such as "happy," "sad," or "angry," etc., the pre-labeled emotion category tags of the multimodal emotion recognition samples are obtained.

[0051] Next, it is checked whether the predicted sentiment category can match the sentiment category label. There are two matching methods: one is an exact match, which requires the predicted sentiment category to be exactly the same as the label. If the match is successful, a higher reward value (e.g., 1.0) is assigned; if the match fails, a lower reward value (e.g., 0.0) is assigned.

[0052] Another approach is inclusion matching, where when a sample is labeled with multiple sentiment categories, a successful match is considered achieved if the predicted sentiment category matches any one of them. If a match is successful, the reward value is determined based on the degree of matching between the predicted sentiment category and the sentiment category label. Specifically, a reward function based on set overlap can be used, with the following calculation formula:

[0053] in, Y represents the reward for the accuracy of predicting the sentiment category of the text, where Y represents the sentiment category label. Indicates the predicted sentiment category, with an upper limit. This prevents the prediction results from becoming excessively inflated. When the number of predicted sentiment categories exceeds this threshold, the reward value is set to 0. The formula means that the reward value equals the ratio of the number of correctly predicted sentiment labels to the number of true labels. Simultaneously, to constrain the model from generating too many irrelevant sentiment labels, the reward value is set to 0 when the number of predicted labels exceeds twice the number of true labels plus one. According to this formula, if the predicted sentiment category perfectly matches the true label, the reward value is 1; if it partially matches, the reward value is the proportion of correct predictions; and if it doesn't match at all, the reward value is 0.

[0054] The technical solution adopted in this embodiment uses the ROUGE index to perform multi-dimensional quantitative evaluation of sentiment interpretation, which can effectively guide the model to generate inference content that is closer to the real interpretation in terms of vocabulary, phrases and sentence structure, and avoid the interpretation content being too simplistic or deviating from the topic; at the same time, the accuracy of sentiment recognition is directly constrained by the sentiment category matching mechanism to ensure that the sentiment category output by the model is correct.

[0055] In one optional embodiment, another implementation method of the two rewards involved in step S150 above is described in detail. This embodiment provides a method for calculating the interpretation perception reward based on sentiment interpretation labels, and a method for calculating the sentiment category accuracy reward based on sentiment wheel mapping. Among them, the sentiment wheel mapping method can well handle sentiment recognition tasks in open vocabulary scenarios and evaluate the semantic similarity between sentiment categories.

[0056] The step S150, "obtaining the interpretation-aware reward of the predicted text based on the sentiment interpretation labels of the multimodal sentiment recognition samples," may include steps C1 to C2: Step C1: Using each ROUGE index in the ROUGE index set, determine the sentiment interpretation label of the multimodal sentiment recognition sample and the score of the predicted sentiment interpretation on that ROUGE index.

[0057] Step C2: The average score of the sentiment interpretation label of the multimodal sentiment recognition sample and the predicted sentiment interpretation on the ROUGE index set is used as the interpretation perception reward of the predicted text.

[0058] In this embodiment, the explanation of steps C1 to C2 can be referred to the explanation of steps A1 to A2 above, and will not be repeated here.

[0059] The step S150, "obtaining the sentiment category accuracy reward of the predicted text based on the sentiment category label of the multimodal sentiment recognition sample and the predicted sentiment category in the predicted text," may include the following steps D1 to D2: Step D1: Map the predicted sentiment category in the predicted text and the sentiment category label of the multimodal sentiment recognition sample to the sectors of the sentiment wheel, respectively.

[0060] The Emotion Wheel is a hierarchical structural model used to organize and represent emotion categories. In the Emotion Wheel, emotion categories are arranged hierarchically from coarse-grained to fine-grained. The central area represents basic emotion categories (such as "happy" and "sad"), while the outer areas represent more fine-grained emotion variations (such as "happy" being further subdivided into "pleasure," "joy," and "ecstasy"). The entire Emotion Wheel is divided into multiple sectors, each corresponding to one or more semantically similar emotion categories.

[0061] In the specific implementation, the first step is to obtain the predicted text... <answer>The predicted sentiment category within the label, as well as the pre-labeled sentiment category labels on the sample. Since the predicted sentiment category in an open-vocabulary scenario might be a word not in the preset list, simple string matching is not feasible. This embodiment uses a sentiment wheel mapping mechanism to map sentiment categories to their corresponding sectors on the sentiment wheel. Specifically, for each sentiment category (whether predicted or labeled), lexical normalization is first performed, and then it is assigned to the corresponding sector through synonym expansion and sentiment wheel hierarchical structure positioning. For example, "joy" and "delight" might be mapped to the sector corresponding to the basic sentiment "happiness".

[0062] Step D2: Detect whether there is semantic overlap between the sector to which the predicted sentiment category is mapped and the sector to which the sentiment category label is mapped, in order to obtain the sentiment category accuracy reward of the predicted text.

[0063] After mapping the predicted sentiment category and sentiment category label to sectors of the sentiment wheel, the system checks for semantic overlap between the sectors they occupy. Semantic overlap can be determined in several ways: one way is to check if they are mapped to the exact same sector. For example, if the predicted "pleasure" and the label "happiness" are both mapped to the "positive sentiment" sector, then overlap exists.

[0064] Another more flexible approach is to detect the hierarchical relationship between sectors. For example, if the predicted "ecstasy" is mapped to a fine-grained sub-region of the "happiness" sector, and the labeled "pleasure" is also mapped to the same fine-grained region, then it is considered to have overlap; or even if they are not in the same fine-grained region, as long as they belong to a coarser-grained upper-level sector, they can also be considered to have partial overlap, and different reward values ​​can be assigned according to the degree of overlap.

[0065] In some embodiments, a sector-complete matching approach is preferred, meaning that semantic overlap is considered to exist only when the predicted sentiment category and the sentiment category label are mapped to the same sector on the sentiment wheel. If semantic overlap exists, a higher accuracy reward (e.g., 1.0) is assigned; if there is no overlap, a lower reward (e.g., 0.0) is assigned.

[0066] For example, the accuracy reward for predicting the sentiment category of text Another way to calculate it can be expressed as:

[0067] Where K represents the number of emotion wheels; This indicates the emotion wheel is k. The introduced hierarchical mapping function will predict the sentiment category. and Emotion Category Tags Mapped to the corresponding sector of the emotion wheel; This indicates the semantic overlap between the sector to which the predicted sentiment category is mapped and the sector to which the sentiment category label is mapped.

[0068] Thus, the emotion wheel evaluation method can effectively handle the emotion recognition problem in open vocabulary scenarios. Even if the emotion category output by the model is not exactly the same as the labeled emotion category, as long as they belong to the same emotion cluster semantically, they can obtain positive rewards, thereby encouraging the model to learn richer and more natural emotion expressions.

[0069] The technical solution adopted in this embodiment uses the ROUGE index to perform multi-dimensional quantitative evaluation of sentiment interpretation, which can effectively guide the model to generate inference content that is closer to the real interpretation in terms of vocabulary, phrase and sentence structure. At the same time, by evaluating the accuracy of sentiment category through the sentiment wheel mapping mechanism, it can handle sentiment recognition tasks in open vocabulary scenarios, so that the model can still obtain reasonable reward signals when faced with diverse sentiment expressions, thereby learning a more generalized sentiment understanding ability.

[0070] In one optional embodiment, for scenarios lacking sentiment explanation labels, a self-evaluation-based method for calculating explanation perception rewards is provided. This embodiment guides the reference model to perform multi-dimensional evaluations of the sentiment explanations generated by the strategy model using preset sentiment explanation evaluation prompts, thereby achieving quantitative rewards for explanation quality even without labeled explanations.

[0071] In practical applications, multimodal sentiment recognition samples may not always contain detailed sentiment interpretation labels. For example, in large datasets, labeling sentiment interpretations is costly. To address this issue, this embodiment utilizes the general evaluation capabilities of a reference model to simulate human evaluation in scoring sentiment interpretations.

[0072] Specifically, the method further includes: evaluating the cue text using sentiment interpretation, and evaluating the quality of the predicted sentiment interpretation through the reference model according to steps E1 to E3 to obtain the interpretation perception reward of the predicted text.

[0073] In this embodiment, a sentiment interpretation evaluation prompt text is designed to guide the reference model in evaluating the predicted sentiment interpretations generated by the policy model. The content of the sentiment interpretation evaluation prompt text could be similar to: "You are a sentiment interpretation evaluation expert. Please rate the following sentiment interpretations from three dimensions: relevance, logical consistency, and contribution." Through this prompt, the reference model is given the role of evaluator, enabling it to perform multi-dimensional evaluations of the input text content based on its pre-trained world knowledge and language understanding capabilities.

[0074] Step E1: Evaluate the correlation between the predicted sentiment interpretation in the predicted text and the predicted sentiment category in the predicted text.

[0075] In this step, the reference model analyzes and predicts sentiment interpretations (e.g.) <think>Does the inference content within the label closely relate to the predicted sentiment category (e.g.)? <answer>Expand on the label "Happy". The evaluation criteria include: whether the emotional cues mentioned in the predicted sentiment interpretation are consistent with the predicted sentiment category, whether the conclusion of the predicted sentiment interpretation naturally leads to that sentiment category, and whether there are descriptions in the predicted sentiment interpretation that are irrelevant to or contradict the final predicted sentiment category.

[0076] For example, if the predicted sentiment category is "sadness," but the predicted sentiment explanation describes "the speaker's lips are upturned and their tone is light," the relevance is low; conversely, if the predicted sentiment explanation describes "the speaker's brows are furrowed and their voice is low," the relevance is high. The reference model assigns a score based on its understanding, such as a score between 1 and 10 or between 0 and 1, as a relevance score.

[0077] Step E2: Evaluate the logical consistency between the different modalities of the predicted sentiment interpretation cues included in the predicted sentiment interpretation.

[0078] In this step, since multimodal sentiment recognition involves multiple information sources such as text, video, and audio, the sentiment interpretation generated by the model should be able to integrate information from these modalities and maintain consistency between the cues from each modality. The reference model checks whether the predicted sentiment interpretation contains descriptions of sentiment cues from multiple modalities, such as facial expressions (video modality), speech tone (audio modality), and word content (text modality), and determines whether there are any contradictions or inconsistencies among these cues.

[0079] For example, if the predicted sentiment explanation describes "facial expression showing anger, but the tone of voice is calm and gentle," there may be logical inconsistency; conversely, if the explanation describes "facial expression showing anger, tone of voice being intense, and wording being aggressive," then the logical consistency is high. The reference model assigns a corresponding score based on the degree of coordination between cues.

[0080] Step E3: Evaluate the contribution of the predicted sentiment interpretation cue of each modality in the predicted sentiment interpretation to the predicted sentiment category in the predicted text.

[0081] In this step, unlike relevance, contribution focuses more on the effectiveness and persuasiveness of the cues, i.e., whether each cue truly provides valuable evidence for the sentiment judgment. The reference model analyzes each modal cue mentioned in the interpretation to determine whether it effectively supports the final sentiment category. For example, for the sentiment category of "happiness," the visual cue "upturned corners of the mouth" contributes more, while the irrelevant cue "background color" contributes less. The reference model comprehensively evaluates the overall contribution of all cues and provides a contribution score.

[0082] After obtaining the scores from the three dimensions mentioned above, they can be fused, for example, by calculating the average or a weighted sum, to obtain the final perceptual reward value. This reward value serves as an alternative in unsupervised scenarios, guiding the model to generate high-quality sentiment interpretations.

[0083] The technical solution adopted in this embodiment, by introducing a self-evaluation mechanism based on prompt text, can effectively supervise the quality of sentiment explanations generated by the model even in the absence of sentiment explanation annotations. This mechanism fully utilizes the language understanding and reasoning capabilities of the reference model itself, comprehensively evaluating the explanations from three key dimensions: relevance, logical consistency, and contribution. This enables the model to learn high-quality sentiment explanations that are closely related to the sentiment category, consistent with multimodal cues, and persuasive during training. Compared to explanation rewards that rely entirely on manual annotation, this embodiment provides a scalable and low-cost alternative, particularly suitable for model training on large-scale datasets or in open-domain scenarios.

[0084] It should be noted that the self-evaluation mechanism provided in this embodiment can complement the aforementioned ROUGE-based supervised evaluation mechanism (i.e., the methods in steps A1 to A2 above), and can be flexibly selected or combined according to data conditions in practical applications.

[0085] In one alternative embodiment, to enhance the model's sensitivity to emotional information, a token-level weighting mechanism based on the emotional dimension is introduced. This mechanism determines the emotional polarity score of each token by evaluating its performance on three emotional dimensions: Valence (V), Arousal (A), and Dominance (D), and uses this score as a weight to weight the loss.

[0086] Specifically, the predicted text and the reference text each include T tokens, where T is an integer greater than 1; the method further includes the following step F1: Step F1: Take t sequentially from 1 to T, and evaluate the t-th word in the predicted text from three dimensions: pleasantness, arousal, and dominance, to determine the sentiment polarity score of the word.

[0087] In this embodiment, the Valence-Arousal-Dominance Lexicon (VAD dictionary) is used as the assessment tool. This dictionary provides standardized VAD scores for a large number of English words. Among them, V indicates the positive or negative nature of the emotion, and the value ranges from negative to positive; Arousal (A) indicates the intensity of the emotion, from calm (low arousal) to excited (high arousal); Dominance (D) indicates the individual's sense of control over the emotional state, from being dominated (low dominance) to being in control (high dominance).

[0088] For the t-th word in the predicted text, a query is performed in the VAD dictionary to obtain its corresponding pleasantness, arousal, and dominance, thus obtaining the sentiment polarity score for that word. For words not included in the dictionary, a default value (such as 0) can be used, or an estimate can be made using the context vector.

[0089] In an optional embodiment, step F1 may specifically include steps F1-1 to F1-3: Step F1-1: For each word in the predicted text, use the pleasure-arousal-dominance dictionary to determine the pleasure, arousal and dominance of the word.

[0090] In this step, the predicted text is segmented into several tokens. For each token, a pleasantness-arousal-dominance dictionary is used to retrieve its corresponding pleasantness, arousal, and dominance. These scores are usually represented as floating-point numbers. For example, for the token "joyful," the dictionary might record V=0.95 (high pleasantness), A=0.80 (high arousal), and D=0.85 (high dominance); for the token "gloomy," it might record V=0.20 (low pleasantness), A=0.35 (low arousal), and D=0.30 (low dominance).

[0091] Step F1-2: Determine the maximum value among the pleasantness, arousal and dominance of the word as the dominant emotional polarity score of the word, and combine the pleasantness, arousal and dominance of the word to obtain the overall emotional polarity score of the word.

[0092] This step quantifies the sentiment salience of lexical units from two perspectives. The dominant sentiment polarity score is the maximum absolute value of the three dimensions, used to capture the strongest sentiment signal from the lexical unit. For example, for the word "joyful," the most prominent value among the three dimensions is the pleasantness score (V=0.95), so the dominant score is 0.95. The overall sentiment polarity score is the overall sentiment intensity obtained by fusing the values ​​of the three dimensions. A root mean square (RMS) fusion method can be used, and this score reflects the overall activity level of the lexical unit in the sentiment space.

[0093] Step F1-3: Obtain the sentiment polarity score of the word based on the dominant sentiment polarity score and the overall sentiment polarity score of the word.

[0094] In this step, the dominant sentiment polarity score and the overall sentiment polarity score are fused to obtain the final sentiment polarity score. The fusion method can be adjusted according to the actual application scenario, for example, linear combination, product fusion, or nonlinear fusion can be used; this embodiment does not limit this.

[0095] For example, taking the linear combination fusion method as an example, the sentiment polarity score of a word can be expressed as:

[0096] in, This represents the sentiment polarity score of the t-th word in the i-th generation; , and Let $\frac{1}{2}$ represent the pleasantness, arousal, and dominance of the $t$-th word element generated in the $i$-th generation, respectively. and These are preset hyperparameters used to balance the contributions of the dominant sentiment polarity score and the overall sentiment polarity score; This represents the root mean square function; This represents the function that maximizes the value. In tasks requiring emphasis on a single, strong emotional dimension, it can be increased... In tasks requiring a comprehensive assessment of overall emotional intensity, it can be increased Through this fusion, the sentiment polarity score reflects both the outstanding performance of a word in a certain dimension and its comprehensive strength across all three dimensions, thus providing a more comprehensive measure of the sentiment salience of a word.

[0097] Furthermore, step S140 above, "obtaining the loss corresponding to the predicted text based on the difference between the predicted text and the reference text," may include the following steps G1 to G3: Step G1: Take t sequentially from 1 to T, and obtain the difference measure of the t-th word in the predicted text based on the difference between the t-th word in the predicted text and the t-th word in the reference text.

[0098] Within the reinforcement learning training framework, it is necessary to measure the difference between the predicted text output by the policy model and the reference text output by the reference model to constrain the update magnitude of the policy model. Specifically, for each position t in the sequence, the difference between the probability distribution of the policy model's output word at that position and the probability distribution of the reference model's output word at the same position is calculated. This difference can be quantified using Kullback-Leibler Divergence (KL Divergence); a larger KL divergence value indicates a greater difference between the two distributions. By iterating through t from 1 to T, the difference metric corresponding to each word position can be obtained, and this difference metric is used to subsequently calculate word-level weights.

[0099] For example, predicting the difference measure of the t-th word in the text. It can be represented as:

[0100] in, Denotes KL divergence, Let represent the model's predicted probability distribution for the t-th word in the i-th generation. It is the expected probability distribution, and G represents the number of generation in the distribution.

[0101] Step G2: Using the sentiment polarity score of the t-th word in the predicted text as the weight, and combining it with the difference measure of the T words in the predicted text, the weighted weight of the t-th word in the predicted text is obtained.

[0102] After obtaining the difference metric and sentiment polarity score for each lexical position, for the t-th lexical, the two are combined to calculate the weighted weight of that lexical in the loss function. This weight will be used to differentiate the loss contribution of each lexical in the subsequent optimization of the objective function.

[0103] The core idea of ​​the weighting mechanism is to assign higher weights to lexical units with higher sentiment polarity scores (i.e., lexical units that carry richer sentiment information), so that the model pays more attention to the generation quality of these sentiment-significant lexical units during the optimization process; while assigning lower weights to lexical units with neutral sentiment or weak sentiment information (such as function words), so as to avoid them having an excessive impact on model training.

[0104] For example, the weighted weight of the t-th word in the i-th generation It can be represented as:

[0105] in, This represents a minimum-maximum normalization function to ensure comparability between different lexical units; To control sparsity parameters, the degree of scaling of lexical importance in this weighting scheme is determined; Let represent the sentiment polarity score of the t-th lexical unit in the i-th generation.

[0106] Step G3: Based on the weighted weight of the t-th word in the predicted text, weight the loss of the t-th word to obtain the loss corresponding to the predicted text.

[0107] After obtaining the weighted weights for each word, these weights are applied to the objective function of reinforcement learning to weight the policy gradient term for each word. Specifically, in optimizing the objective function, the loss contribution of each word is composed of its weighted weight multiplied by the corresponding probability ratio and advantage value, and the update magnitude is limited by the Clip-Higher policy. The Clip-Higher policy expands the clipping range by introducing an upper bound, thereby relaxing the restriction on increasing the probability of low-frequency words, helping to reduce the bias towards high-frequency words and promoting the generation of more diverse predicted text.

[0108] For example, optimize the objective function Represented as:

[0109] in, represents the expectation operation; q represents the multimodal sentiment recognition sample; a represents the label corresponding to the multimodal sentiment recognition sample q (such as sentiment category label, sentiment interpretation label, etc.); D represents the distribution of the training dataset; Let represent the predicted text generated in the i-th iteration; G represents the number of predicted texts generated by the policy model to be trained based on the multimodal sentiment recognition sample q. This represents the old policy model before the update (i.e., the policy before the update), which is used to sample and generate predicted text; This represents the weighted weight of the t-th word in the i-th generation; Represents the probability ratio, used to measure the updated policy. Compared to the previous strategy Deviation between; This represents the total reward after normalization. c represents the function that takes the minimum value; This represents the clipping function. and These are the lower and upper limits for clipping, respectively; This represents the parameters of the policy model to be trained.

[0110] The technical solution adopted in this implementation introduces a word-level weighting mechanism based on the emotional dimensions of pleasantness, arousal, and dominance, achieving focused optimization of emotionally significant words. On the one hand, this mechanism makes the model pay more attention to words carrying strong emotional information during training, thereby improving the accuracy of emotion recognition; on the other hand, by focusing on key emotional words in the emotion explanation part, the model is guided to generate more accurate and emotionally rich explanations. Compared with the traditional loss calculation method that treats all words equally, this embodiment provides a more refined training method that is more in line with the laws of emotion cognition, effectively improving the model's sensitivity and expressive ability to emotional information.

[0111] like Figure 2 As shown, Figure 2 This is an overall architecture diagram of a training method for a multimodal emotion recognition model provided in this application embodiment. Specifically, a pre-trained multimodal large language model is initialized to obtain a reference model and a policy model to be trained; multimodal emotion recognition samples (text, video, audio) are input into the reference model to obtain reference text, and multimodal emotion recognition samples are input into the policy model to be trained to obtain predicted text; wherein, there can be multiple predicted texts, and for each predicted text, a corresponding reward (such as interpretation perception reward, emotion category accuracy reward, standard format reward, etc.) is calculated, and the rewards are fused to obtain the total reward corresponding to the predicted text; furthermore, based on the differences between the predicted text and each reference text, the loss corresponding to each predicted text is obtained; finally, based on the total reward and the loss corresponding to the predicted text, the parameters of the multimodal large language model to be trained are updated.

[0112] There are two calculation methods for the explanation-perceived reward. One is based on sentiment explanation labels: using each ROUGE index in the ROUGE index set, the sentiment explanation label of the multimodal sentiment recognition sample and the score of the predicted sentiment explanation on that ROUGE index are determined; the average score of the sentiment explanation label and the predicted sentiment explanation on the ROUGE index set is then used as the explanation-perceived reward for the predicted text. The other is based on a self-evaluation mechanism: using the sentiment explanation evaluation prompt text, the reference model is guided to score the sentiment explanation generated by the strategy model in multiple dimensions (relevance, logical consistency, and contribution) to obtain the explanation-perceived reward for the predicted text.

[0113] There are two methods for calculating the sentiment category accuracy reward. One method is based on sentiment category matching: it detects whether the predicted sentiment category in the predicted text can match the sentiment category label of the multimodal sentiment recognition sample to obtain the sentiment category accuracy reward of the predicted text. The other method is based on sentiment wheel mapping: it maps the predicted sentiment category in the predicted text and the sentiment category label of the multimodal sentiment recognition sample to sectors of the sentiment wheel; thus, it detects whether there is semantic overlap between the two sectors to obtain the sentiment category accuracy reward of the predicted text.

[0114] Furthermore, to enhance the model's sensitivity to sentiment information, a word-level weighting mechanism based on the sentiment dimension can be introduced. This mechanism evaluates the performance of each word on three sentiment dimensions—pleasure, arousal, and dominance—to determine its sentiment polarity score. This score is then used as a weight to combine with the word's difference metric to weight the loss of that word, thereby obtaining the loss corresponding to the predicted text.

[0115] Thus, this method improves the performance of multimodal emotion recognition models in terms of both emotion recognition accuracy and emotion interpretation quality by introducing interpretation perception rewards and emotion category accuracy rewards, and combining the difference loss between the policy model and the reference model to jointly optimize the model within the reinforcement learning framework.

[0116] The training method of the multimodal emotion recognition model of this application will be illustrated below with two specific embodiments.

[0117] Example 1: Example 1 describes an Emotion Polarity Weighted Scheme (Emotion Reasoner). This scheme calculates the perceptual reward based on emotion interpretation labels, calculates the emotion category accuracy reward based on the emotion wheel mapping, and employs a word-level weighting mechanism based on the emotion dimension to calculate the loss corresponding to the predicted text. (Refer to...) Figure 3 As shown, Figure 3 This is a flowchart illustrating the steps of an emotion polarity weighting scheme provided in an embodiment of this application. The scheme includes the following steps: Step S310: Obtain multimodal emotion recognition samples, wherein the modalities of the multimodal emotion recognition samples include at least two of the following modalities: text modality, video modality, and audio modality.

[0118] Step S320: Initialize the pre-trained multimodal large language model to obtain the reference model and the policy model to be trained.

[0119] Step S330: Input the multimodal emotion recognition sample into the policy model to be trained to obtain the predicted text, and input the multimodal emotion recognition sample into the reference model to obtain the reference text.

[0120] Step S340: The predicted text and the reference text each include T lexical units; t is sequentially selected from 1 to T, and the difference measure of the t-th lexical unit in the predicted text is obtained based on the difference between the t-th lexical unit in the predicted text and the t-th lexical unit in the reference text; the sentiment polarity score of the t-th lexical unit in the predicted text is used as the weight, combined with the difference measure of the T lexical units in the predicted text, to obtain the weighted weight of the t-th lexical unit in the predicted text; based on the weighted weight of the t-th lexical unit in the predicted text, the loss of the t-th lexical unit is weighted to obtain the loss corresponding to the predicted text.

[0121] Step S350: Using each ROUGE index in the ROUGE index set, determine the sentiment interpretation label of the multimodal sentiment recognition sample and the score of the predicted sentiment interpretation on the ROUGE index; take the average score of the sentiment interpretation label of the multimodal sentiment recognition sample and the predicted sentiment interpretation on the ROUGE index set as the interpretation perception reward of the predicted text.

[0122] Step S360: Map the predicted sentiment category in the predicted text and the sentiment category label of the multimodal sentiment recognition sample to sectors of the sentiment wheel respectively; detect whether there is semantic overlap between the sector to which the predicted sentiment category is mapped and the sector to which the sentiment category label is mapped, so as to obtain the sentiment category accuracy reward of the predicted text.

[0123] Step S370: The interpretation perception reward and sentiment category accuracy reward corresponding to the predicted text are fused to obtain the total reward corresponding to the predicted text.

[0124] Step S380: Based on the total reward corresponding to the predicted text and the loss corresponding to the predicted text, update the parameters of the policy model to be trained to obtain a multimodal sentiment recognition model.

[0125] The technical solution provided in this embodiment optimizes the training of a multimodal emotion recognition model through an emotion polarity weighting mechanism. It employs an emotion category accuracy reward based on emotion wheel mapping, mapping predicted and actual emotion categories to sectors of the emotion wheel for semantic overlap detection, effectively handling emotion recognition tasks in open-vocabulary scenarios. Furthermore, it uses an explanation-aware reward based on the ROUGE metric, quantitatively evaluating the model-generated emotion explanations from multiple dimensions—vocabulary, phrase, and sentence structure—guiding the model to generate reasoning content closer to the true explanations. This reward mechanism effectively avoids overly simplistic or off-topic explanations, ensuring both accuracy and clarity. In addition, by introducing a word-level weighting mechanism based on three emotion dimensions—pleasure, arousal, and dominance—the model pays closer attention to the generation quality of these key emotion words during training. Compared to traditional loss calculation methods that treat all words equally, this embodiment provides a more refined training method that better aligns with the laws of emotion cognition, effectively improving the model's sensitivity and expressive ability to emotional information. Thus, the multimodal emotion recognition model trained in this embodiment can output accurate and interpretable emotion analysis results, which can be widely used in scenarios that require emotion understanding and transparent decision-making, such as human-computer interaction, mental health monitoring, and education, and has significant technological progress and practical value.

[0126] Example 2: Example 2 describes a self-evaluation mechanism scheme (EmotionReasoner-SelfEval). This scheme uses a self-evaluation mechanism to calculate the explanatory perceived reward, uses an emotion wheel mapping to calculate the emotion category accuracy reward, and uses a word-level weighting mechanism for the emotion dimension to calculate the loss corresponding to the predicted text. (Refer to...) Figure 4 As shown, Figure 4 This is a flowchart illustrating the steps of a self-assessment mechanism scheme provided in an embodiment of this application. The scheme includes the following steps: Step S410: Obtain multimodal emotion recognition samples, wherein the modalities of the multimodal emotion recognition samples include at least two of the following modalities: text modality, video modality, and audio modality.

[0127] Step S420: Initialize the pre-trained multimodal large language model to obtain the reference model and the policy model to be trained.

[0128] Step S430: Input the multimodal emotion recognition sample into the policy model to be trained to obtain the predicted text, and input the multimodal emotion recognition sample into the reference model to obtain the reference text.

[0129] Step S440: The predicted text and the reference text each include T lexical units; t is sequentially selected from 1 to T, and the difference measure of the t-th lexical unit in the predicted text is obtained based on the difference between the t-th lexical unit in the predicted text and the t-th lexical unit in the reference text; the sentiment polarity score of the t-th lexical unit in the predicted text is used as the weight, combined with the difference measure of the T lexical units in the predicted text, to obtain the weighted weight of the t-th lexical unit in the predicted text; based on the weighted weight of the t-th lexical unit in the predicted text, the loss of the t-th lexical unit is weighted to obtain the loss corresponding to the predicted text.

[0130] Step S450: Evaluate the prompt text using sentiment interpretation. Evaluate the quality of the predicted sentiment interpretation using the reference model according to the following steps to obtain the interpretation perception reward of the predicted text: evaluate the relevance between the predicted sentiment interpretation in the predicted text and the predicted sentiment category in the predicted text; evaluate the logical consistency between the predicted sentiment interpretation cues of different modalities included in the predicted sentiment interpretation; evaluate the contribution of the predicted sentiment interpretation cues of each modality in the predicted sentiment interpretation to the predicted sentiment category in the predicted text.

[0131] Step S460: Map the predicted sentiment category in the predicted text and the sentiment category label of the multimodal sentiment recognition sample to sectors of the sentiment wheel respectively; detect whether there is semantic overlap between the sector to which the predicted sentiment category is mapped and the sector to which the sentiment category label is mapped, so as to obtain the sentiment category accuracy reward of the predicted text.

[0132] Step S470: The interpretation perception reward and sentiment category accuracy reward corresponding to the predicted text are fused to obtain the total reward corresponding to the predicted text.

[0133] Step S480: Based on the total reward corresponding to the predicted text and the loss corresponding to the predicted text, update the parameters of the policy model to be trained to obtain a multimodal sentiment recognition model.

[0134] The technical solution provided in this embodiment achieves model optimization training in the absence of sentiment annotations through a self-evaluation mechanism. In practical applications, large-scale sentiment datasets often lack detailed sentiment annotation labels, resulting in high annotation costs. This self-evaluation mechanism, based on prompt text, leverages the language understanding and reasoning capabilities of the reference model to automatically evaluate sentiment interpretations from three key dimensions: relevance, logical consistency, and contribution. This allows for a quantitative reward of interpretation quality even without labeled annotations. This mechanism provides a scalable and low-cost alternative, particularly suitable for model training on large-scale datasets or in open-domain scenarios. By retaining the sentiment category accuracy reward based on the sentiment wheel mapping and the word-level weighting mechanism based on the sentiment dimension, the model's sentiment recognition performance is still guaranteed. Thus, the multimodal sentiment recognition model trained in this embodiment can still output high-quality sentiment interpretations and accurate sentiment recognition results even in the absence of sentiment annotations, providing a feasible technical path for large-scale practical applications and possessing significant theoretical and practical value.

[0135] To illustrate the effectiveness of the technical solutions provided in the embodiments of this application, the performance of the method of this application on multimodal emotion recognition tasks is experimentally verified and explained below.

[0136] This experiment evaluated the results on nine publicly available multimodal emotion recognition datasets, covering three categories of tasks: fine-grained emotion recognition, basic emotion recognition, and emotion analysis. Statistical information for the MER dataset is shown in Table 1.

[0137] Table 1: Statistical information of the MER dataset

[0138] (1) Main experimental results: The performance of the sentiment polarity weighting scheme and self-evaluation mechanism provided in this application is compared with that of baseline models (Qwen2.5-Omni-7B, AffectGPT) on nine datasets. As shown in Table 2, the experimental results show that the sentiment polarity weighting scheme achieves the best performance on all nine datasets, with an average accuracy of 84.54%, an improvement of 10.33% compared to AffectGPT. The self-evaluation mechanism scheme achieves an average accuracy of 81.48%, an improvement of 7.27% compared to AffectGPT. On basic sentiment recognition tasks (MER2023, MER2024, MELD, IEMOCAP), the sentiment polarity weighting scheme achieves a significant improvement of 15%-18%, indicating that the method in this application has good generalization ability. On the fine-grained sentiment recognition task (OV-MERD), the sentiment polarity weighting scheme improves by 5.96%, verifying the robustness of the method in open vocabulary scenarios.

[0139] Table 2 compares the performance of each model on the multimodal emotion recognition dataset.

[0140] (2) Interpretability assessment results: Interpretability was evaluated on the OV-MERD dataset using five key metrics: cue overlap, label overlap, multimodal sentiment evidence reasoning evaluation (MEER-E), user trust evaluation (UTAI-E), and logical clarity evaluation (LACE-E). As shown in Table 3, experimental results demonstrate that the self-evaluation mechanism outperforms all baseline models on fine-grained interpretability metrics (MEER-E, UTAI-E, and LACE-E), with an average score of 8.19, a 1.18 improvement over AffectGPT. Particularly on UTAI-E and LACE-E, the self-evaluation mechanism achieves scores of 8.39 and 8.94, respectively, indicating that the sentiment interpretations generated by our method are more practical and understandable. The self-evaluation mechanism also outperforms the baselines on cue overlap and label overlap metrics, achieving scores of 7.88 and 5.59, respectively, demonstrating its advantage in identifying sentiment cues and matching sentiment labels.

[0141] Table 3. Interpretability evaluation results of each model on the OV-MERD dataset.

[0142] (3) Ablation experiments with reward design: Three ablation experiments were designed and compared with Qwen2.5-Omni-7B. Ablation experiment 1 used standard format rewards and calculated accuracy rewards based on sentiment category matching (step B1); ablation experiment 2 used standard format rewards, calculated interpreted perceived rewards based on sentiment interpretation labels (steps A1 to A2), and calculated accuracy rewards based on sentiment category matching (step B1); ablation experiment 3 used standard format rewards, calculated interpreted perceived rewards based on sentiment interpretation labels (steps A1 to A2), and calculated accuracy rewards based on sentiment wheel mapping (steps D1 to D1).

[0143] As shown in Table 4, the experimental results indicate that introducing format and accuracy rewards (ablation experiment 1) improved MER accuracy by 6.69%, but decreased the fine-grained explanatory index by 1.26%. Adding an explanatory perception reward (ablation experiment 2) improved the fine-grained explanatory index by 0.63 while maintaining MER accuracy, effectively mitigating the trade-off between accuracy and interpretability. Using an emotion wheel accuracy reward (ablation experiment 3) further improved MER accuracy to 81.19%, while simultaneously increasing the fine-grained explanatory index to 7.66, achieving the optimal balance.

[0144] Table 4 Ablation experiment results with different reward designs

[0145] (4) Ablation experiment of training strategy: This experiment uses Rouge+OV (calculating interpretability reward based on sentiment interpretation labels and calculating sentiment category accuracy reward based on sentiment wheel mapping) as the baseline, and compares it with the sentiment polarity weighting scheme and self-evaluation mechanism scheme provided in this application. As shown in Table 5, the experimental results show that the sentiment polarity weighting scheme achieves a MER accuracy of 84.54%, an improvement of 3.35% compared to the baseline, verifying the effectiveness of the word-level weighting mechanism proposed in this application. Although the self-evaluation mechanism scheme sacrifices slightly in MER accuracy, it performs best in interpretability metrics, especially reaching 8.19 in the fine-grained interpretability metric, verifying its effectiveness in unlabeled scenarios.

[0146] Table 5 Ablation Experiment Results of Different Training Strategies

[0147] In summary, the technical solutions provided in this application have achieved excellent performance on multiple public datasets: Regarding sentiment recognition accuracy, the sentiment polarity weighting scheme provided in this application achieves an average accuracy of 84.54% on nine datasets, significantly outperforming existing methods. Regarding sentiment interpretation quality, the self-evaluation mechanism provided in this application achieves a score of 8.19 on the fine-grained interpretability index, verifying the effectiveness of the method in generating high-quality sentiment interpretations. Regarding component effectiveness, ablation experiments verify the contributions of key components such as interpretable perceived reward, sentiment wheel mapping, and sentiment polarity weighting to model performance. The method in this application demonstrates excellent performance across multiple tasks, including basic sentiment recognition, fine-grained sentiment recognition, and sentiment analysis, showcasing good generalization ability.

[0148] The above experimental results show that the technical solution provided in this application can effectively solve the problem of low quality of sentiment interpretation in the prior art. The trained multimodal sentiment recognition model can output sentiment analysis results that are both accurate and interpretable, which has significant technical progress and practical value.

[0149] This application also provides a training device for a multimodal emotion recognition model, referring to... Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a training device for a multimodal emotion recognition model provided in an embodiment of this application. The device includes: The sample acquisition module 510 is used to acquire multimodal emotion recognition samples, wherein the modalities of the multimodal emotion recognition samples include at least two of the following modalities: text modality, video modality, and audio modality; Initialization module 520 is used to initialize the pre-trained multimodal large language model to obtain the reference model and the policy model to be trained; The sample input module 530 is used to input the multimodal emotion recognition sample into the policy model to be trained to obtain predicted text, and to input the multimodal emotion recognition sample into the reference model to obtain reference text; The loss calculation module 540 is used to obtain the loss corresponding to the predicted text based on the difference between the predicted text and the reference text. The reward calculation module 550 is used to obtain the interpretation perception reward corresponding to the predicted text based on the sentiment interpretation label of the multimodal sentiment recognition sample, and to obtain the sentiment category accuracy reward corresponding to the predicted text based on the sentiment category label of the multimodal sentiment recognition sample and the predicted sentiment category in the predicted text. The reward fusion module 560 is used to fuse the interpretation perception reward and the sentiment category accuracy reward corresponding to the predicted text to obtain the total reward corresponding to the predicted text. The parameter update module 570 is used to update the parameters of the policy model to be trained based on the total reward and the loss corresponding to the predicted text, so as to obtain a multimodal emotion recognition model.

[0150] It is understood that the training device for the multimodal emotion recognition model in the embodiments of this application can implement the training method for the multimodal emotion recognition model in the above embodiments. The training device for the multimodal emotion recognition model has the same advantages as the training method for the multimodal emotion recognition model in the prior art, and will not be repeated here.

[0151] This application also provides an electronic device, see embodiments thereof. Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps of the training method for the multimodal emotion recognition model described in the embodiments of this application.

[0152] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the training method for the multimodal emotion recognition model described in this application.

[0153] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the training method for the multimodal emotion recognition model described in this application.

[0154] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0155] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0159] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0160] The training method, apparatus, device, and medium for a multimodal emotion recognition model provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / answer> < / think> < / answer> < / answer> < / think> < / answer> < / think> < / answer> < / think>

Claims

1. A training method for a multimodal emotion recognition model, characterized in that, The method includes: Obtain multimodal emotion recognition samples, wherein the modalities of the multimodal emotion recognition samples include at least two of the following modalities: text modality, video modality, and audio modality; The pre-trained multimodal large language model is initialized to obtain a reference model and a policy model to be trained; The multimodal emotion recognition samples are input into the policy model to be trained to obtain predicted text, and the multimodal emotion recognition samples are input into the reference model to obtain reference text; The loss corresponding to the predicted text is obtained based on the difference between the predicted text and the reference text; Based on the sentiment interpretation labels of the multimodal sentiment recognition samples, the interpretation perception reward corresponding to the predicted text is obtained; and based on the sentiment category labels of the multimodal sentiment recognition samples and the predicted sentiment category in the predicted text, the sentiment category accuracy reward corresponding to the predicted text is obtained. The total reward corresponding to the predicted text is obtained by fusing the interpretation perception reward and the sentiment category accuracy reward corresponding to the predicted text. Based on the total reward and loss corresponding to the predicted text, the parameters of the policy model to be trained are updated to obtain a multimodal emotion recognition model.

2. The training method for the multimodal emotion recognition model according to claim 1, characterized in that, Based on the sentiment interpretation labels of the multimodal sentiment recognition samples, the interpretation-aware reward of the predicted text is obtained, including: Using each ROUGE index in the ROUGE index set, determine the sentiment interpretation label of the multimodal sentiment recognition sample and the score of the predicted sentiment interpretation on that ROUGE index; The average score of the sentiment interpretation label of the multimodal sentiment recognition sample and the predicted sentiment interpretation on the ROUGE index set is used as the interpretation perception reward of the predicted text. Based on the sentiment category labels of the multimodal sentiment recognition samples and the predicted sentiment category in the predicted text, a sentiment category accuracy reward for the predicted text is obtained, including: The predicted sentiment category in the predicted text is checked to see if it matches the sentiment category label of the multimodal sentiment recognition sample, and the sentiment category accuracy reward of the predicted text is obtained.

3. The training method for the multimodal emotion recognition model according to claim 1, characterized in that, Based on the sentiment interpretation labels of the multimodal sentiment recognition samples, the interpretation-aware reward of the predicted text is obtained, including: Using each ROUGE index in the ROUGE index set, determine the sentiment interpretation label of the multimodal sentiment recognition sample and the score of the predicted sentiment interpretation on that ROUGE index; The average score of the sentiment interpretation label of the multimodal sentiment recognition sample and the predicted sentiment interpretation on the ROUGE index set is used as the interpretation perception reward of the predicted text. Based on the sentiment category labels of the multimodal sentiment recognition samples and the predicted sentiment category in the predicted text, a sentiment category accuracy reward for the predicted text is obtained, including: The predicted sentiment category in the predicted text and the sentiment category label of the multimodal sentiment recognition sample are respectively mapped to sectors of the sentiment wheel; Detect whether there is semantic overlap between the sector to which the predicted sentiment category is mapped and the sector to which the sentiment category label is mapped, in order to obtain the sentiment category accuracy reward of the predicted text.

4. The training method for the multimodal emotion recognition model according to claim 1, characterized in that, The method further includes: The quality of the predicted sentiment interpretation is evaluated using the reference model through the following steps to obtain the perceived reward of the predicted text: Evaluate the correlation between the predicted sentiment interpretation in the predicted text and the predicted sentiment category in the predicted text; Evaluate the logical consistency among the different modalities of predicted sentiment interpretation cues included in the predicted sentiment interpretation; Evaluate the contribution of the predicted sentiment interpretation cue of each modality in the predicted sentiment interpretation to the predicted sentiment category in the predicted text.

5. The training method for the multimodal emotion recognition model according to any one of claims 1-4, characterized in that, The predicted text and the reference text each include T tokens, where T is an integer greater than 1; the method further includes: The t-th word in the predicted text is evaluated from 1 to T in sequence, based on three dimensions: pleasantness, arousal, and dominance, to determine the sentiment polarity score of that word. The loss corresponding to the predicted text is obtained based on the difference between the predicted text and the reference text, including: By taking t sequentially from 1 to T, the difference measure of the t-th word in the predicted text is obtained based on the difference between the t-th word in the predicted text and the t-th word in the reference text. The weighted weight of the t-th word in the predicted text is obtained by using the sentiment polarity score of the t-th word in the predicted text as the weight and combining it with the difference measure of the T words in the predicted text. Based on the weighted weight of the t-th word in the predicted text, the loss of the t-th word is weighted to obtain the loss corresponding to the predicted text.

6. The training method for the multimodal emotion recognition model according to claim 5, characterized in that, The t-th lexical unit in the predicted text is evaluated from three dimensions: pleasantness, arousal, and dominance, to determine the sentiment polarity score of that lexical unit, including: For each word element in the predicted text, the pleasantness, arousal, and dominance of that word element are determined using a pleasantness-arousal-dominance dictionary. The maximum value among the pleasantness, arousal and dominance of the word element is determined as the dominant emotional polarity score of the word element. The overall emotional polarity score of the word element is obtained by combining the pleasantness, arousal and dominance of the word element. The emotional polarity score of a word is obtained based on its dominant emotional polarity score and its overall emotional polarity score.

7. A training device for a multimodal emotion recognition model, characterized in that, The device includes: The sample acquisition module is used to obtain multimodal emotion recognition samples, wherein the modalities of the multimodal emotion recognition samples include at least two of the following modalities: text modality, video modality, and audio modality; The initialization module is used to initialize the pre-trained multimodal large language model to obtain the reference model and the policy model to be trained. The sample input module is used to input the multimodal emotion recognition sample into the policy model to be trained to obtain predicted text, and to input the multimodal emotion recognition sample into the reference model to obtain reference text; The loss calculation module is used to obtain the loss corresponding to the predicted text based on the difference between the predicted text and the reference text. The reward calculation module is used to obtain the interpretation perception reward corresponding to the predicted text based on the sentiment interpretation label of the multimodal sentiment recognition sample, and to obtain the sentiment category accuracy reward corresponding to the predicted text based on the sentiment category label of the multimodal sentiment recognition sample and the predicted sentiment category in the predicted text. The reward fusion module is used to fuse the interpretation perception reward and the sentiment category accuracy reward corresponding to the predicted text to obtain the total reward corresponding to the predicted text. The parameter update module is used to update the parameters of the policy model to be trained based on the total reward and the loss corresponding to the predicted text, so as to obtain a multimodal sentiment recognition model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the training method for the multimodal emotion recognition model according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the training method for the multimodal emotion recognition model according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the training method for the multimodal emotion recognition model as described in any one of claims 1-6.

Images