Multimodal sentiment reasoning method based on acoustic and facial feature alignment
By introducing verifiable reward reinforcement learning based on facial action units and acoustic features, the problem of neglecting fine-grained cues and relying on manual annotation in emotion recognition of large multimodal models is solved, and efficient and interpretable emotion reasoning analysis is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-19
- Publication Date
- 2026-07-03
AI Technical Summary
Existing multimodal large models tend to overlook fine-grained acoustic and visual cues when performing emotion recognition, and rely on costly manually labeled data, resulting in a lack of interpretability and accuracy in the emotion reasoning process.
We employ reinforcement learning (RLVR) with verifiable rewards to fine-tune a large multimodal model, introducing facial action units and acoustic features. Through an automated process, we extract and anchor these features into the emotional context, construct an instruction dataset, and perform model fine-tuning and self-correction under weak supervision.
Without relying on large-scale manual annotation, it significantly improves the accuracy and interpretability of sentiment reasoning, reduces visual illusions and prosodic deafness, and generates sentiment reasoning analysis supported by objective evidence.
Smart Images
Figure CN122334466A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing and sentiment computing, specifically a method for fine-tuning a multimodal large language model and interpretable multimodal sentiment inference based on acoustic and facial feature alignment. Background Technology
[0002] Affective computing is undergoing a paradigm shift from closed-set classification to interpretable multimodal emotion recognition (EMER). With the rapid development of large multimodal language models (MLLMs), emotion recognition systems can not only predict discrete emotion tags but also articulate the reasoning process leading to that conclusion in natural language. This interpretability is crucial for applications requiring high levels of trust and empathy, such as psychological counseling and human-computer interaction.
[0003] While current multimodal large models demonstrate powerful semantic understanding and language generation capabilities, they still face challenges in fine-grained perceptual alignment. Existing general-purpose large models, when performing sentiment reasoning, often rely more on high-level semantic intuition and less on explicitly anchoring their reasoning process to objective physical evidence. This can cause the models to overlook subtle paralinguistic acoustic cues such as tone variations and speech rate when processing complex emotions, or to produce biases in describing visual details that do not match actual physical actions.
[0004] Furthermore, to enable large models to accurately align raw multimodal signals with detailed objective evidence, high-quality supervised datasets are typically required. However, manually identifying fine-grained facial movements and analyzing acoustic features by experts is time-consuming and labor-intensive, making large-scale scaling difficult. This results in a general lack of fine-grained interpretive annotations in existing large-scale sentiment datasets. Therefore, guiding models to automatically focus on and utilize objective features to generate sentiment reasoning analysis supported by objective physical evidence, without relying on large-scale, detailed manual annotation, has become a pressing area for optimization in the field of sentiment computing. Summary of the Invention
[0005] The purpose of this invention is to provide a multimodal emotion reasoning method based on acoustic and facial feature alignment, addressing the shortcomings of existing technologies. It employs reinforcement learning (RLVR) with verifiable rewards. This invention effectively solves the problems of large-scale model visual illusions, prosodic deafness, and high dependence on manual annotation, enabling the model to perform objective evidence-based reasoning without large-scale human supervision.
[0006] The specific technical solution for achieving the objective of this invention is as follows:
[0007] A multimodal emotion reasoning method based on acoustic and facial feature alignment employs reinforcement learning (RLVR) with verifiable rewards to fine-tune a large multimodal model. Objective physical cues, namely facial action units and acoustic features, are explicitly introduced into the reasoning chain of the large multimodal model to reduce visual illusions and prosodic deafness, generating emotion reasoning analysis supported by objective evidence. The method specifically includes the following steps:
[0008] Step 1: Multimodal Feature Alignment and Enhancement
[0009] An automated process is used to extract objective facial action units (AUs) and acoustic feature levels from raw videos. Then, an inference chain is synthesized using a teacher language model to anchor the facial action unit AUs and acoustic feature levels to the emotional context, constructing an instruction dataset with both facial and acoustic features—the instruction fine-tuning dataset. and large-scale weakly supervised datasets ;
[0010] Step 2: Cold Start Fine-Tuning of Evidence Perception
[0011] Fine-tuning the dataset using instructions Supervised fine-tuning of large multimodal language models; initializing the model's reasoning ability so that it outputs structured objective evidence before outputting natural language reasoning results; given instructions. Multimodal input , Indicates video input. Indicates audio input, target sequence joint probability Decomposed into:
[0012]
[0013] The multimodal large model is generated in the order of structured evidence → inference chain → final prediction; whereby, This is a JSON-formatted sequence containing extracted AUS and acoustic levels. To explain the inferential text that links features with emotions, As the final emotional label;
[0014] Optimize model parameters through standard supervised fine-tuning. Minimize the negative log-likelihood loss function of the structured sequence:
[0015]
[0016] in, The parameters of a multimodal large language model are the model weights that need to be optimized during training. This indicates that the instruction fine-tunes the dataset, which includes multimodal inputs. Pairing data with the desired target sequence; This represents the mathematical expectation, i.e., the expectation across the entire instruction fine-tuning dataset. The average loss calculated above; Represents the target sequence The total length; This indicates the time step or index position in the sequence of the current generation step; Indicates the target sequence, the first... The real word element at each position; Indicates the first All historical word sequences that have been generated before this position; Indicates that given model parameters Multimodal input ,instruction and prepositional sequences Under these conditions, the model predicts the next correct word. The conditional probability;
[0017] By forcing the generation order This forces the model to first identify physical features, then synthesize reasoning, and finally draw conclusions, thereby reducing illusions and enhancing interpretability;
[0018] Step 3: Reinforcement learning based on verifiable rewards
[0019] On large-scale datasets lacking human-annotated reasoning, a group-relative strategy optimization algorithm is used to self-correct the model; the extraction results of facial feature extraction tools and speech recognition tools are introduced as evaluation criteria to construct a multi-dimensional verifiable reward function; and illusory outputs that do not match physical features are penalized to force the model to use correct facial and audio features for emotion reasoning.
[0020] Furthermore, step 1 specifically includes:
[0021] 1-1: Visual Action Unit Extraction: The frame-level intensity of dozens of facial action units was extracted using the facial feature extraction tool Py-Feat. The average score in the temporal dimension was calculated to generate a video-level representation. The top 5 facial action units with scores greater than 0.5 were selected and mapped to semantic descriptions.
[0022] 1-2: Acoustic Level Extraction: Word-level timestamps are provided by the speech recognition tool Whisper to separate human voice segments, and six core acoustic indicators are extracted: RMS energy, dynamic range, mean fundamental frequency, fundamental frequency range, speech rate, and pause rate; continuous acoustic features are discretized into five levels based on quintiles and mapped to text descriptions;
[0023] 1-3: Data Construction: Using the teacher's language model, the original dataset's real sentiment labels, extracted AUSs, acoustic levels, and inference text were input to synthesize a fine-tuned instruction dataset. and large-scale weakly supervised datasets .
[0024] Furthermore, step 3 specifically includes:
[0025] 3-1: For each query From the current strategy Medium sampling Output groups The update objective of the model policy is to maximize the following objective function:
[0026]
[0027] in, This indicates the current strategy, i.e., the model parameters of the multimodal large language model that is being updated; This represents the old strategy, i.e., the model parameters of the model used to sample the input query before the last update iteration; Representing mathematical expectation, covering data from the dataset. Mid-sampled query , and the output group generated by the old strategy; This represents a large-scale weakly supervised dataset used in the reinforcement learning phase; Indicates targeting the same query The number of output groups sampled by the model; Indicates the index of the sampled output group; This is the group-normalized dominance function; This represents the KL divergence relative to the reference model, used to ensure training stability; This indicates that the current model performs well in a given query. In this case, generate specific output The probability of; This indicates that the old model is in response to a given query. In this case, generate specific output The probability of the two is used to calculate the importance sampling ratio. This represents a reference strategy, used to provide a baseline during reinforcement learning to prevent the model from deviating too far from its initial language generation capabilities after updates. The penalty coefficient representing the KL divergence is used to control the tolerance of the model to the difference from the reference model when optimizing in the direction of high reward.
[0028] 3-2: Design a comprehensive multi-dimensional total reward function:
[0029]
[0030] in:
[0031] Accuracy Rewards The value is 1 when the predicted label equals the true label, and 0 otherwise.
[0032] Format rewards Used to penalize structural errors and verify that the output contains valid JSON and the specified XML tag format;
[0033] AU Matching Rewards : Calculate the model-predicted AUS using F1 scores ( ) and the real objective AUS extracted by the tool ( Alignment between )
[0034]
[0035] Acoustic consistency reward The exact match rate between the discrete acoustic levels predicted by the calculation model and the results extracted by the tool;
[0036] 3-3: If the model claims that the target person is performing a certain action or making a certain sound, but the feature extraction tool does not detect it, and This will reduce the likelihood of hallucinations, thus suppressing their generation in subsequent iterations.
[0037] The beneficial effects of this invention are:
[0038] To address the technical shortcomings of existing large-scale models in multimodal sentiment analysis, such as "visual illusion" and "prosodic deafness" due to their tendency to detach from actual input features, this invention creatively introduces objective facial action units (AUs) and acoustic-level features as verification benchmarks, and combines them with a reinforcement learning framework with verifiable rewards to guide the model in self-correction. This mechanism effectively overcomes the heavy reliance of traditional sentiment recognition models on large-scale, high-cost manually labeled data, enabling efficient fine-tuning of the model with minimal computational cost. Furthermore, this invention anchors the implicit reasoning process of large models to quantifiable facial and biometric evidence, significantly improving the final accuracy of sentiment recognition in complex scenarios and greatly enhancing the interpretability of the sentiment reasoning path and the logical reliability of the results, thus possessing extremely high practical application value. Attached Figure Description
[0039] Figure 1 This is a flowchart of the present invention;
[0040] Figure 2This is an example diagram illustrating the reasoning performance of the fine-grained emotion model of this invention. Detailed Implementation
[0041] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.
[0042] The present invention provides a multimodal emotion reasoning method based on acoustic and facial feature alignment, which specifically includes the following steps:
[0043] Step 1: Multimodal Feature Alignment and Enhancement: Utilizing an automated process, objective facial action units (AUs) and acoustic feature levels are extracted from the raw video. These physical features are then anchored to the emotional context through a teacher language model-synthesized inference chain, constructing a command dataset with a rigorous objective basis. The extraction and processing of multimodal features specifically includes:
[0044] Visual action unit extraction: Frame-level intensity of 20 facial action units is extracted using feature extraction tools (such as Py-Feat), the average score in the temporal dimension is calculated to generate a video-level representation, and the top 5 AUs with scores greater than 0.5 are selected and mapped to semantic descriptions.
[0045] Acoustic level extraction: Word-level timestamps are provided using speech recognition tools (such as Whisper) to separate human voice segments, and six core acoustic metrics are extracted: RMS energy, dynamic range, mean fundamental frequency, fundamental frequency range, speech rate, and pause rate. Continuous acoustic features are discretized into five levels based on quintiles and mapped to text descriptions.
[0046] Data Construction: Utilizing a teacher language model, inputting real sentiment labels, extracted AUS, acoustic levels, and original inference text, a high-quality instruction fine-tuning dataset with rigorous logic is synthesized. and large-scale weakly supervised datasets .
[0047] Step 2, Cold Start Fine-tuning of Evidence Awareness: Utilizing the Enhanced Instruction Dataset Explained Above Supervised fine-tuning of a large multimodal language model. Initialize the model's reasoning ability so that it outputs structured objective evidence before outputting natural language reasoning results. Given multimodal input... and instructions target sequence The joint probability decomposition form is:
[0048]
[0049] The model must be generated in the order of "structured evidence → inference chain → final prediction". This is a JSON-formatted sequence containing extracted AUS and acoustic levels. To explain the inferential text that links features with emotions, For the final emotional label.
[0050] Optimize model parameters using standard supervised fine-tuning (SFT). Minimize the negative log-likelihood loss function of the structured sequence:
[0051]
[0052] By forcing the generation order This forces the model to first identify physical features, then synthesize reasoning, and finally draw conclusions, thereby reducing illusions and enhancing interpretability.
[0053] Step 3: Reinforcement Learning Based on Verifiable Rewards: On large-scale datasets lacking human-annotated reasoning, the Group Relative Policy Optimization (GRPO) algorithm is used to self-correct the model. Objective analysis tools are introduced as evaluation criteria, and a multi-dimensional verifiable reward function is constructed to penalize illusory outputs that do not conform to physical characteristics, forcing the model to base its reasoning on verifiable objective evidence.
[0054] For each query From the current strategy Medium sampling Output groups The update objective of the model policy is to maximize the following objective function:
[0055]
[0056] in, This is the group-normalized dominance function; This represents the KL divergence relative to the reference model, used to ensure training stability.
[0057] Design a comprehensive multi-dimensional total reward function:
[0058]
[0059] in:
[0060] Accuracy Bonus ( ): The value is 1 when the predicted label is equal to the true label, and 0 otherwise.
[0061] Format rewards ( ): Used to penalize structural errors and verify that the output contains valid JSON and the specified XML tag format.
[0062] AU Matching Rewards ( ): Calculate the model-predicted AUS using the F1 score ( ) and the real objective AUS extracted by the tool ( Alignment between )
[0063]
[0064] Acoustic consistency reward ( ): The exact match rate between the discrete acoustic levels predicted by the calculation model and the results extracted by the tool.
[0065] Through this mechanism, if the model claims that the target person is "frowning" (AU4) but the feature extraction tool does not detect eyebrow activity, This will reduce the likelihood of such hallucinations in subsequent iterations.
[0066] Example
[0067] See Figure 1 In this embodiment, training and deployment are carried out using the HumanOmni-0.5B multimodal large model and the Open-R1 reinforcement learning framework.
[0068] 1) Data preparation stage for multimodal feature alignment and enhancement:
[0069] First, to construct a high-quality cold start instruction dataset, this embodiment uses the MER2025 sentiment dataset. For the videos in the dataset, the Py-Feat facial expression analysis tool is used to extract the frame-level intensity of 20 facial action units (AUs) of the characters in the videos, calculate the average score in the time dimension, and select the top 5 AUs with scores greater than 0.5 as dominant visual cues (e.g., mapping them to text descriptions such as "AU4: frowning" and "AU5: upper eyelid lifting").
[0070] In terms of acoustics, the Whisper-large-v3 speech recognition model was used to transcribe the audio and extract word-level timestamps to accurately separate human voice from background noise. Subsequently, on the separated human voice segments, the Librosa audio processing library was used to calculate six acoustic metrics: RMS energy, dynamic range, mean fundamental frequency, fundamental frequency range, speech rate, and pause rate. These continuous values were divided into five discrete levels based on the quintiles of the dataset and mapped to natural language descriptions ranging from "very low" to "very high".
[0071] Next, the extracted objective features (AUs and acoustic levels), real sentiment labels, and the original inference text are input into the DeepSeek-V3 large language model, which synthesizes a logically rigorous explanatory inference text. After screening, a high-quality instruction fine-tuning dataset (i.e., cold start dataset) containing approximately 350 samples is finally constructed.
[0072] Furthermore, for subsequent reinforcement learning, this embodiment selected two large-scale in-the-wild sentiment datasets, DFEW and MAFW. Only the aforementioned automated feature extraction process (obtaining feature JSON) was performed on these data, without generating synthetic inference text, thus constructing a large-scale weakly supervised dataset.
[0073] 2) The cold start fine-tuning phase of evidence perception:
[0074] HumanOmni-0.5B, which has native audio-video interleaving capabilities, was selected as the base model. Supervised fine-tuning (SFT) was performed on it using a dataset of approximately 350 high-quality instructions constructed using MER2025.
[0075] During fine-tuning, given a multimodal input and instructions target sequence The joint probability decomposition form is:
[0076]
[0077] in, This is a JSON-formatted sequence containing extracted AUS and acoustic levels. To explain the inferential text that links features with emotions, For the final emotional label.
[0078] Optimize model parameters using standard supervised fine-tuning (SFT). Minimize the negative log-likelihood loss function of the structured sequence:
[0079]
[0080] During training, the model's generation format was strictly constrained, forcing it to first output structured evidence in JSON format containing objective agent labels, then output the natural language inference chain contained within the "think" label, and finally output the sentiment label prediction result contained within the "answer" label. Through this stage, the model initially mastered the ability to first list evidence and then perform inference.
[0081] 3) Reinforcement learning phase based on verifiable rewards:
[0082] Based on the SFT model, this embodiment uses the Group Relative Policy Optimization (GRPO) algorithm for reinforcement learning to eliminate visual illusions or prosodic deafness that may occur when the model generates descriptions.
[0083] This phase utilizes a large-scale weakly supervised dataset composed of DFEW and MAFW for training. Using the Open-R1 framework, in each iteration, the model generates multiple sets of inference outputs for the input audiovisual data. The system extracts the structured features from the outputs and compares them with objective benchmark data extracted beforehand using Py-Feat and Librosa. Specifically, the matching degree between the model-predicted AUs and the tool-extracted AUs is calculated using the F1 score as a visual reward, and the acoustic level matching degree is calculated using the accuracy matching rate as an acoustic reward. Combining the format reward and the accuracy reward of the final label forms a comprehensive reward signal to guide the model policy update.
[0084] For each query From the current strategy Medium sampling Output groups The update objective of the model policy is to maximize the following objective function:
[0085]
[0086] in, This is the group-normalized dominance function; This represents the KL divergence relative to the reference model, used to ensure training stability.
[0087] After this self-correction phase, a finely tuned multimodal sentiment reasoning model is finally obtained. When faced with input from entirely new environments, this model can not only accurately identify emotions but also output a fine-grained sentiment reasoning analysis report that fully conforms to objective physical facts. (See also...) Figure 2The model receives video information and can accurately identify whether the emotion of the person in the video is happiness. It also outputs a detailed reasoning process and fine-grained feature indicators as supporting evidence. Specifically, the model intuitively displays the extracted structured facial action units (AU01 inner eyebrow lift, AU17 chin push, AU24 lip press, AU01 inner eyebrow lift) and acoustic feature parameters (extremely high average pitch, extremely low intonation fluctuation, low speaking volume, extremely low speaking speed, medium volume dynamic range, and total pause duration) at the output. Meanwhile, the model's thought process—"In the video, a female character is in an indoor environment. Her facial expressions—smiling eyes, upturned corners of the mouth but downturned lips, especially the movement of the pucker-lip muscle (AU15)—initially suggest she is having a pleasant conversation or has received good news. As the scene progresses, her gaze shifts downwards while her smile remains evident, perhaps indicating a change in topic or contemplation; at the end, she closes her eyes and blinks rapidly, suggesting a possible shift in emotions towards a more positive state, such as excitement or joy. Although the audio tone is calm and the emotional descriptions in the text are neutral or even negative, failing to provide strong evidence for the underlying emotions..." However, the subtle movement of the inner brow lift (AU01) further maintained the calm state of the brow, supporting the inference that she maintained a positive attitude throughout. Although the extremely low speech rate and moderate pitch variation, among other acoustic features, did not explicitly point to a single emotional state, but may reflect a restrained and reflective vocal quality, combined with visual cues including specific facial movements and expressions, it can still be inferred that she likely entered a more positive emotional state after her attention was diverted. This clearly presents a complete chain of reasoning: how to capture subtle movements of a person's facial muscles and combine them with vocal characteristics to ultimately infer the person's emotional state. This output method based on objective evidence not only provides accurate emotional labels but also offers a transparent and traceable chain of physical evidence, significantly enhancing the interpretability and reliability of the emotion recognition results.
[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A multi-modal sentiment reasoning method based on acoustic and facial feature alignment, characterized in that, The method specifically includes the following steps: Step 1: Multimodal Feature Alignment and Enhancement Objective facial action units AUs and acoustic feature levels are extracted from the original video using an automated process; the facial action units AUs and acoustic feature level features are anchored into the emotional context through a teacher language model synthesis inference chain to construct an instruction dataset with facial and acoustic features, namely an instruction fine-tuning dataset and a large-scale weakly supervised dataset Step 2: Cold Start Fine-Tuning of Evidence Perception Fine-tuning the dataset using instructions Supervised fine-tuning of large-scale multimodal language models; initializing the model's reasoning ability so that it outputs structured objective evidence before outputting natural language reasoning results; given instructions. Multimodal input , Indicates video input. Indicates audio input, target sequence joint probability Decomposed into: The multimodal large model is generated in the order of structured evidence → inference chain → final prediction; whereby, This is a JSON-formatted sequence containing extracted AUS and acoustic levels. To explain the inferential text that links features with emotions, As the final emotional label; Optimize model parameters through standard supervised fine-tuning. Minimize the negative log-likelihood loss function of the structured sequence: in, The parameters of a multimodal large language model are the model weights that need to be optimized during training. This indicates that the instruction fine-tunes the dataset, which includes multimodal inputs. Pairing data with the desired target sequence; This represents the mathematical expectation, i.e., the expectation across the entire instruction fine-tuning dataset. The average loss calculated above; Represents the target sequence The total length; This indicates the time step or index position in the sequence of the current generation step; Indicates the target sequence, the first... The real word element at each position; Indicates the first All historical word sequences that have been generated before this position; Indicates that given model parameters Multimodal input ,instruction and prepositional sequences Under these conditions, the model predicts the next correct word. The conditional probability; By forcing the generation order This forces the model to first identify physical features, then synthesize reasoning, and finally draw conclusions, thereby reducing illusions and enhancing interpretability; Step 3: Reinforcement learning based on verifiable rewards On large-scale datasets lacking human-annotated reasoning, a group-relative strategy optimization algorithm is used to self-correct the model; the extraction results of facial feature extraction tools and speech recognition tools are introduced as evaluation criteria to construct a multi-dimensional verifiable reward function; and illusory outputs that do not match physical features are penalized to force the model to use correct facial and audio features for emotion reasoning.
2. The multimodal emotion reasoning method according to claim 1, characterized in that, Step 1 specifically includes: 1-1: Visual Action Unit Extraction: The frame-level intensity of dozens of facial action units was extracted using the facial feature extraction tool Py-Feat. The average score in the temporal dimension was calculated to generate a video-level representation. The top 5 facial action units with scores greater than 0.5 were selected and mapped to semantic descriptions. 1-2: Acoustic Level Extraction: Word-level timestamps are provided by the speech recognition tool Whisper to separate human voice segments, and six core acoustic indicators are extracted: RMS energy, dynamic range, mean fundamental frequency, fundamental frequency range, speech rate, and pause rate; continuous acoustic features are discretized into five levels based on quintiles and mapped to text descriptions; 1-3: Data Construction: Using the teacher's language model, the original dataset provides real sentiment labels, extracted AUSs, acoustic levels, and inference text, and an instruction fine-tuning dataset is synthesized. and large-scale weakly supervised datasets .
3. The multimodal emotion reasoning method according to claim 1, characterized in that, Step 3 specifically includes: 3-1: For each query From the current strategy Medium sampling Output groups The update objective of the model policy is to maximize the following objective function: in, This indicates the current strategy, i.e., the model parameters of the multimodal large language model that is being updated; This represents the old strategy, i.e., the model parameters of the model used to sample the input query before the last update iteration; Representing mathematical expectation, covering data from the dataset. Mid-sampled query , and the output group generated by the old strategy; This represents a large-scale weakly supervised dataset used in the reinforcement learning phase; Indicates targeting the same query The number of output groups sampled by the model; Indicates the index of the sampled output group; This is the group-normalized dominance function; This represents the KL divergence relative to the reference model, used to ensure training stability; This indicates that the current model performs well in a given query. In this case, generate specific output The probability of; This indicates that the old model is in response to a given query. In this case, generate specific output The probability of the two is used to calculate the importance sampling ratio. This represents a reference strategy, used to provide a baseline during reinforcement learning to prevent the model from deviating too far from its initial language generation capabilities after updates. The penalty coefficient representing the KL divergence is used to control the tolerance of the model to the difference from the reference model when optimizing in the direction of high reward. 3-2: Design a comprehensive multi-dimensional total reward function: Among them: accuracy reward The value is 1 when the predicted label equals the true label, and 0 otherwise; Format reward. Used to penalize structural errors and verify that the output contains valid JSON and the specified XML tag format; AU matching reward. : Calculate the model-predicted AUS using F1 scores ( ) and the real objective AUS extracted by the tool ( Alignment between ) Acoustic consistency reward The exact match rate between the discrete acoustic levels predicted by the calculation model and the results extracted by the tool; 3-3: If the model claims that the target person is performing a certain action or making a certain sound, but the feature extraction tool does not detect it, and This will reduce the likelihood of hallucinations, thus suppressing their generation in subsequent iterations.