A brain-inspired open-vocabulary multimodal emotion recognition method

Abstract

This invention discloses a brain-inspired open-vocabulary multimodal emotion recognition method, belonging to the field of multimodal emotion computing technology. The method includes: acquiring audio, video, and text data of the object to be recognized and extracting features to obtain acoustic, visual, and semantic features, mapping them to form multimodal features; constructing an emotion cue dataset MER-CLUE; using the LoRA method to adjust the parameters of the cue generation base language model to obtain a cue generation large language model; generating cue hints using the cue generation large language model; training the emotion reasoning base language model for emotion reasoning; generating open-vocabulary emotion descriptions based on the emotion reasoning large language model and outputting the emotion recognition result. This invention achieves unified multimodal emotion expression and generates interpretable open-vocabulary emotion descriptions; combined with optimization, it improves the accuracy, generalization, and training efficiency of emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of multimodal emotion computing technology, and in particular to a brain-inspired open-vocabulary multimodal emotion recognition method. Background Technology

[0002] With the development of artificial intelligence technology, affective computing has gradually evolved from emotion recognition in a single voice or image modality to multimodal emotion understanding. Existing emotion recognition methods typically rely on pre-defined emotion labeling systems, classifying features from single modalities such as audio, images, or text. This makes it difficult to accurately characterize the multidimensional, fine-grained, and context-dependent aspects of human emotions. Furthermore, traditional models have limited ability to model the causes of emotions, changes in emotional states, and the correlations between different modalities in complex scenarios, resulting in poor generalization and insufficient interpretability of emotion recognition results.

[0003] In recent years, Large Language Models (LLMs) have demonstrated powerful semantic reasoning capabilities in natural language understanding and generation tasks. However, directly applying LLMs to multimodal emotion recognition still faces many challenges: First, audio, visual, and semantic features are difficult to represent in a unified space, making it difficult for the model to fully capture cross-modal relationships between different modalities; second, traditional classification-based emotion labeling systems limit the model's expressive range, preventing it from outputting emotion explanations in open-ended vocabulary and natural language forms; third, the lack of effective cross-modal emotion cue supervision mechanisms makes it difficult for LLMs to learn fine-grained human understanding of the causes, expressions, and details of emotions.

[0004] Furthermore, training a complete large language model is costly and involves a huge number of parameters, which is not conducive to efficient customized optimization for specific emotion recognition tasks. Although efficient parameter fine-tuning techniques such as LoRA can reduce training costs, how to combine multimodal emotion supervision signals to enable large language models to have both interpretive emotion description generation capabilities and stable emotion classification capabilities remains a key problem that current technologies have not been able to effectively solve.

[0005] Therefore, there is an urgent need for a multimodal emotion reasoning method that can integrate multimodal features in a unified emotional semantic space, and enable large language models to acquire fine-grained emotion reasoning capabilities through emotion cue generation and supervision guidance, and output open-vocabulary emotion descriptions and emotion recognition results. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the problem to be solved by this invention is how to endow the computational model with explicit reasoning ability similar to the human brain in the multimodal emotion recognition task of open vocabulary, so that it can generate recognition results that are accurate, fine-grained in emotion description and interpretable and traceable in decision-making process for a given audio, video and text input, while significantly improving the accuracy of open vocabulary emotion recognition.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] In a first aspect, embodiments of the present invention provide a brain-inspired open-vocabulary multimodal emotion recognition method, comprising: acquiring audio data, video data and text data of the object to be recognized and extracting features to obtain acoustic features, visual features and semantic features, and mapping the acoustic features, visual features and semantic features to a unified emotional semantic space to form multimodal features;

[0010] Based on the audio and video data of the objects to be identified, an emotion cue dataset MER-CLUE is constructed;

[0011] The LoRA method is used to adjust the parameters of the preset cue generation base large language model, and the cue generation base large language model is optimized based on the sentiment cue dataset MER-CLUE to obtain the cue generation large language model.

[0012] Based on the big language model for cue generation, audio cue text and video cue text are generated from the audio data and video data of the object to be identified, respectively, and then concatenated to form a cue prompt. The cue prompt and multimodal features are used as the inference input of the big language model for cue generation.

[0013] Based on reasoning input containing clues, the LoRA method is used to train the sentiment reasoning base big language model for sentiment reasoning;

[0014] Based on the sentiment reasoning big language model trained by sentiment reasoning, open-vocabulary sentiment descriptions are generated, and finally sentiment recognition results are output.

[0015] As a preferred embodiment of the brain-inspired open-vocabulary multimodal emotion recognition method of the present invention, the acoustic features, visual features, and semantic features are mapped to a unified emotional semantic space to form multimodal features, including:

[0016] The audio data x of the input object to be identified are respectively... a Video data x v and text data x t Feature extraction is performed, and acoustic feature vectors are obtained using an audio encoder. Visual feature vectors are obtained using a video encoder. Semantic feature vectors are obtained using a text encoder. ;

[0017] acoustic feature vectors Mapping to the sentiment semantic space yields an acoustic sentiment embedding representation. ; Visual feature vector Mapping to the sentiment semantic space yields a visual sentiment embedding representation. ; to transform text feature vectors Mapping to the sentiment semantic space yields a semantic sentiment embedding representation. ;

[0018] Embedding the acoustic emotion representation Visual emotion embedding representation and semantic sentiment embedding representation Alignment and integration are carried out in a unified emotional semantic space.

[0019] As a preferred embodiment of the brain-inspired open-vocabulary multimodal emotion recognition method of the present invention, wherein: based on the audio and video data of the object to be identified, an emotion cue dataset MER-CLUE is constructed, including:

[0020] The audio and video data of the object to be identified are input into a preset cue generation base big language model, which generates audio emotional cue text and video emotional cue text related to the emotional state. The generated audio emotional cue text and video emotional cue text are collected and stored to form the emotional cue dataset MER-CLUE, which is used to describe multimodal emotional sensory evidence.

[0021] As a preferred embodiment of the brain-inspired open-vocabulary multimodal emotion recognition method described in this invention, the method involves optimizing the cue generation base language model based on the MER-CLUE emotion cue dataset to obtain a cue generation language model, including:

[0022] The formal descriptions of audio and video emotion cues corresponding to the large language model for cue generation are as follows:

[0023] ;

[0024] ;

[0025] in, It is an acoustic emotion embedding representation. It is a visual emotion embedding representation. It is a semantic sentiment embedding representation. It is a semantic feature vector. It refers to the cue descriptions generated by the cue generation base large language model. LLM is a pre-defined cue generation base large language model. It is the cross-entropy loss that ensures the generated text matches the cues in MER-CLUE. It is the acoustic emotion embedding representation of the i-th sample. It is the visual sentiment embedding representation of the i-th sample. It is the semantic sentiment embedding representation of the i-th sample. It is the semantic feature vector of the i-th sample. Let be the emotional cue text corresponding to the i-th sample, N be the total number of samples, and logP(|) be the log probability;

[0026] After training, the resulting large language model with fine-grained cue generation capabilities is solidified into a large language model for cue generation.

[0027] As a preferred embodiment of the brain-inspired open-vocabulary multimodal emotion recognition method of the present invention, wherein: audio cue text and video cue text are generated from the audio data and video data of the object to be recognized, respectively, and concatenated to form cue prompts; the cue prompts and multimodal features are used as inference inputs for the cue generation large language model, including:

[0028] Audio emotional cue text and video emotional cue text are generated using a large language model for cue generation, respectively. The audio emotional cue text describes acoustic emotional evidence such as pitch changes, energy fluctuations, and speech rate changes, while the video emotional cue text describes visual emotional evidence such as local facial expressions, posture dynamics, and movement amplitude.

[0029] The audio emotional cue text and the video emotional cue text are concatenated in a preset order to construct a cue prompt containing cross-modal emotional evidence;

[0030] The aforementioned clues and multimodal features are input into the clue generation language model, enabling the model to generate emotional description text in open-vocabulary form during the reasoning stage by simultaneously utilizing the fused sensory cue text and low-level multimodal evidence.

[0031] As a preferred embodiment of the brain-inspired open-vocabulary multimodal emotion recognition method of the present invention, wherein: generating emotion description text in open-vocabulary form includes:

[0032] The overall generation process is formalized as follows:

[0033] ;

[0034] in, It is all the text in the inference text describing the true sentiment of the i-th sample. LLM is a pre-defined cue generation base large language model. It is an acoustic emotion embedding representation. It is a visual emotion embedding representation. It is a semantic sentiment embedding representation. is the semantic feature vector, and c is the context condition;

[0035] The training objective is formalized as follows:

[0036] ;

[0037] in, It is a token or the entire sequence in the true sentiment description text of the i-th sample. It is the acoustic emotion embedding representation of the i-th sample. It is the visual sentiment embedding representation of the i-th sample. It is the semantic sentiment embedding representation of the i-th sample. It is the semantic feature vector of the i-th sample. Here, is the context condition for the i-th sample, N is the total number of samples, and logP(|) is the log probability. It is the optimization goal.

[0038] As a preferred embodiment of the brain-inspired open-vocabulary multimodal emotion recognition method described in this invention, the method includes: training an emotion reasoning base large language model using the LoRA method based on reasoning input containing clues, including: performing low-rank decomposition and injecting trainable parameters into the weight matrix of the emotion reasoning base large language model, including the attention mechanism layer and / or feedforward network layer, using the LoRA method, and updating the newly added low-rank parameters with gradients; calculating and outputting the emotion classification loss for the emotion reasoning base large language model according to the emotion category labeled in the samples, and using the emotion classification loss to supervise the optimization of the LoRA parameters.

[0039] As a preferred embodiment of the brain-inspired open-vocabulary multimodal emotion recognition method described in this invention, the method generates open-vocabulary emotion descriptions based on an emotion reasoning large language model trained on emotion reasoning, and finally outputs emotion recognition results, including:

[0040] The emotional reasoning big language model is used to generate open-vocabulary emotional description text under the joint conditions of multimodal features and clue prompts. The emotional description text is used to characterize the fine-grained emotional state of the input sample in a natural language manner. The emotional description text is semantically parsed or classified and mapped to obtain the corresponding emotional recognition result. The emotional recognition result includes a preset emotional category, emotional intensity or emotional label.

[0041] In a second aspect, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, they implement the steps of a brain-inspired open-vocabulary multimodal emotion recognition method as described in the first aspect of the present invention.

[0042] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, they implement the steps of a brain-inspired open-vocabulary multimodal emotion recognition method as described in the first aspect of the present invention.

[0043] The beneficial effects of this invention are as follows: By introducing the sentiment cue generation and sentiment inference mechanism of a large language model, this invention achieves fine-grained understanding of multimodal emotional states. The large language model is optimized based on MER-CLUE-based supervised signals, enabling the model to generate interpretable open-vocabulary sentiment descriptions, significantly improving the expressive and generalization abilities of sentiment recognition. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0047] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0048] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0049] Example 1

[0050] Reference Figure 1 This is the first embodiment of the present invention, which provides a brain-inspired open-vocabulary multimodal emotion recognition method, comprising:

[0051] S101: Acquire audio data, video data, and text data of the object to be identified and extract features to obtain acoustic features, visual features, and semantic features. Map the acoustic features, visual features, and semantic features to a unified emotional semantic space to form multimodal features.

[0052] Furthermore, the audio data x of the input object to be identified is processed separately. a Video data x v and text data x t Feature extraction is performed, and acoustic feature vectors are obtained using an audio encoder. Visual feature vectors are obtained using a video encoder. Semantic feature vectors are obtained using a text encoder. ;

[0053] acoustic feature vectors Mapping to the sentiment semantic space yields an acoustic sentiment embedding representation. ; Visual feature vector Mapping to the sentiment semantic space yields a visual sentiment embedding representation. ; to transform text feature vectors Mapping to the sentiment semantic space yields a semantic sentiment embedding representation. ;

[0054] Embedding the acoustic emotion representation Visual emotion embedding representation and semantic sentiment embedding representation Alignment and integration are carried out in a unified emotional semantic space.

[0055] S102: Based on the audio and video data of the object to be identified, construct the emotion cue dataset MER-CLUE.

[0056] Furthermore, the audio and video data of the object to be identified are input into a preset cue generation base big language model, which generates audio emotional cue text and video emotional cue text related to the emotional state. The generated audio emotional cue text and video emotional cue text are collected and stored to form the emotional cue dataset MER-CLUE, which is used to describe multimodal emotional sensory evidence.

[0057] It should be noted that in the video portion, the Qwen2.5-VL-7B-Instruct model is first used to analyze the video under preset cue constraints, generating textual cues describing subtle emotion-related signals. These cues include, but are not limited to, facial expression changes, eye movement features, body posture, speech tone changes, and environmental context. The generated cue text does not directly provide an emotion category judgment, but rather serves as an intermediate semantic representation for multimodal emotion understanding. The video cue text and audio cue text are then organized and labeled to construct the emotion cue dataset MER-CLUE. This dataset serves as the first-stage training data for the cue generation foundational language model, enabling it to generate fine-grained emotion cue descriptions from multimodal features.

[0058] In the audio portion, the Kimi-Audio-7B-Instruct model is employed. Structured sentiment cues are used to guide the model in analyzing speech features from phonetic and psychological perspectives. These cues constrain the model to focus on acoustic indicators such as pitch, volume, speech rate, intelligibility, breathing patterns, vocal cord tension, and pauses. The model is guided to establish relationships between these acoustic indicators and emotional state, emotional intensity, and valence, thereby generating acoustic sentiment cue text for emotion understanding. This cue text does not directly provide an emotion category judgment but serves as an intermediate semantic representation in the multimodal emotion reasoning process.

[0059] The Emotional Cue Dataset MER-CLUE contains 28,194 multimodal samples with structured annotations, including primary and secondary emotions, facial, body, and acoustic cues, as well as quantified intensity levels. Its broad coverage, with an average of 730 words per sample and over 35,000 unique emotional terms, makes MER-CLUE a high-quality resource for interpretable OV-MER research.

[0060] S103: The LoRA method is used to adjust the parameters of the preset cue generation base language model, and the cue generation base language model is optimized based on the sentiment cue dataset MER-CLUE to obtain the cue generation language model.

[0061] Furthermore, the audio and video emotion cue descriptions corresponding to the large language model for cue generation are constructed, and their formal expression is as follows:

[0062] ;

[0063] ;

[0064] in, It is an acoustic emotion embedding representation. It is a visual emotion embedding representation. It is a semantic sentiment embedding representation. It is a semantic feature vector. It refers to the cue descriptions generated by the cue generation base large language model. LLM is a pre-defined cue generation base large language model. It is the cross-entropy loss that ensures the generated text matches the cues in MER-CLUE. It is the acoustic emotion embedding representation of the i-th sample. It is the visual sentiment embedding representation of the i-th sample. It is the semantic sentiment embedding representation of the i-th sample. It is the semantic feature vector of the i-th sample. Let be the emotional cue text corresponding to the i-th sample, N be the total number of samples, and logP(|) be the log probability;

[0065] After training, the resulting large language model with fine-grained cue generation capabilities is solidified into a large language model for cue generation.

[0066] It should be noted that during the optimization process, the audio, video, and fusion projector parameters remain trainable. Simultaneously, a LoRA parameter fine-tuning strategy is employed for the large language model to enhance its ability to generate cue descriptions from sensory features. After a predetermined number of training rounds, the resulting model is called the sentiment cue generation model, which can generate fine-grained audio and video cue texts based on multimodal input. This sentiment cue generation model is used in the subsequent open-vocabulary sentiment reasoning training stage based on cue cues (i.e., stage two), providing interpretable intermediate cue representations for this stage, thereby simulating the temporary storage and encoding functions of human working memory in the process of emotional cognition.

[0067] S104: Based on the cue-generating large language model, audio cue text and video cue text are generated from the audio data and video data of the object to be identified, respectively, and then concatenated to form a cue prompt. The cue prompt and multimodal features are used as the inference input of the cue-generating large language model.

[0068] Furthermore, an audio emotional cue text and a video emotional cue text are generated using a cue generation large language model, respectively. The audio emotional cue text describes acoustic emotional evidence such as pitch changes, energy fluctuations, and speech rate changes, while the video emotional cue text describes visual emotional evidence such as local facial expressions, posture dynamics, and movement amplitude.

[0069] The audio emotional cue text and the video emotional cue text are concatenated in a preset order to construct a cue prompt containing cross-modal emotional evidence;

[0070] The aforementioned clues and multimodal features are input into the clue generation language model, enabling the model to generate emotional description text in open-vocabulary form during the reasoning stage by simultaneously utilizing the fused sensory cue text and low-level multimodal evidence.

[0071] Furthermore, the overall generation process is formalized as follows:

[0072] ;

[0073] in, It is all the text in the inference text describing the true sentiment of the i-th sample. LLM is a pre-defined cue generation base large language model. It is an acoustic emotion embedding representation. It is a visual emotion embedding representation. It is a semantic sentiment embedding representation. is the semantic feature vector, and c is the context condition;

[0074] The training objective is formalized as follows:

[0075] ;

[0076] in, It is a token or the entire sequence in the true sentiment description text of the i-th sample. Acoustic emotion embedding representation of the i-th sample It is the visual sentiment embedding representation of the i-th sample. It is the semantic sentiment embedding representation of the i-th sample. It is the semantic feature vector of the i-th sample. Here, is the context condition for the i-th sample, N is the total number of samples, and logP(|) is the log probability. It is the optimization goal.

[0077] S105: Based on reasoning input containing clues, the LoRA method is used to train the sentiment reasoning base large language model for sentiment reasoning.

[0078] Furthermore, the LoRA method is used to perform low-rank decomposition and inject trainable parameters into the weight matrix of the sentiment reasoning base language model, including the attention mechanism layer and / or feedforward network layer, and to update the gradient of the newly added low-rank parameters; the sentiment classification loss is calculated for the sentiment reasoning base language model according to the sentiment category of the sample label, and the sentiment classification loss is output to supervise the optimization of LoRA parameters.

[0079] It should be noted that in this embodiment, after fusing multimodal features and cue vectors, the fused multimodal input sequence is fed into the sentiment reasoning-based large language model for supervised fine-tuning: low-rank decomposition is performed to introduce trainable low-rank parameters; training samples with sentiment labels are constructed, each training sample containing: image region features, scene features, fine-grained cue features; text description or dialogue content; cue vectors; and target sentiment label.

[0080] The sentiment reasoning foundation big language model generates a predicted sentiment distribution after receiving a multimodal input sequence. Using cross-entropy loss, during backpropagation, only the LoRA low-rank matrix is ​​updated, while the other parameters of the sentiment reasoning foundation big language model remain frozen. The updated LoRA parameters gradually learn the influence mechanism of image cues and text cues on sentiment, thereby significantly improving the model's reasoning ability.

[0081] After training, a large-scale emotional reasoning language model is obtained. During the reasoning phase, the multimodal features corresponding to the object to be identified, along with clues output by the emotional cue generation model, are used as input to the model. Without human intervention, the large-scale emotional reasoning language model, based on its learned multimodal emotional reasoning capabilities, automatically outputs emotional description text in open-vocabulary form through autoregressive text generation. Furthermore, based on the emotional semantics contained in the emotional description text, the emotional state is analyzed to obtain the corresponding emotional classification result.

[0082] S106: Based on the sentiment reasoning big language model trained by sentiment reasoning, it generates open-vocabulary sentiment descriptions and finally outputs sentiment recognition results.

[0083] Furthermore, the emotional reasoning big language model is used to generate open-vocabulary emotional description text under the joint conditions of multimodal features and clue prompts. The emotional description text is used to characterize the fine-grained emotional state of the input sample in a natural language manner. The emotional description text is then subjected to semantic parsing or classification mapping to obtain the corresponding emotional recognition result. The emotional recognition result includes a preset emotional category, emotional intensity, or emotional label.

[0084] It should be noted that in this training phase, the MER-Caption+ dataset is used as the training sample. Clues and multimodal features are introduced to train the sentiment reasoning big language model, enabling the sentiment reasoning big language model to learn to make comprehensive inferences based on sentiment cues and output open-word sentiment descriptions and sentiment recognition results.

[0085] Example 2

[0086] This embodiment is the second embodiment of the present invention, which provides a brain-inspired open-vocabulary multimodal emotion recognition method. In order to verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0087] The experimental setup followed the MER-FG track requirements of MER2025, with adaptive adjustments made to the dataset partitioning. Specifically, in the MER-Caption+ dataset, 90% of the samples (28,194) were selected as the training set, and the remaining 10% (3,133) were used as the test set. Additionally, the OV-MERD dataset was introduced as a separate test set to evaluate the model's cross-dataset generalization ability in open-word sentiment recognition tasks. Finally, the model performance was evaluated on both the MER-Caption+ and OV-MERD test sets. Since the label space is open and multi-labeled, the predicted results and ground truth labels were first determined through lexical reconstruction. Merge with synonyms Standardization is performed. Based on Prachk's emotion wheel, emotion words are mapped to a coarse-grained layer. and fine-grained layers To reduce the bias caused by a single division, a set of five emotion wheels was used. For each emotion cycle and layers Calculate the precision, recall, and F1 score at the sample level, denoted as . The final average of the coarse and fine particle size fractions is calculated as follows:

[0088] ;

[0089] in, It is a comprehensive evaluation index at the k-th emotion granularity level, where N is the total number of samples, j is the sample index, and k is the emotion granularity level index.

[0090] Table 1. Comparison of S1 and S2 scores on OV-MERD and MER-Caption+

[0091]

[0092] Where S1 is the macro-average S1 evaluation index of the model at the coarse-grained emotion level, S2 is the macro-average S2 evaluation index of the model at the fine-grained emotion level, and avg is the average of S1 and S2.

[0093] Table 1 shows the results on MER-Caption+ and OVMERD. The API-based models (ChatGPT-4o, Gemini2.5Pro) scored the lowest, with an average of less than 41% on MER-Caption+. This is mainly because they cannot handle audio modalities and can only rely on text and visual information. In contrast, open-source multimodal models (Phi-4-MM

[18] , Qwen2.5-Omni-7B

[19] ) can integrate video and audio cues, which makes them significantly better on both datasets. Sensitivity-specific methods showed stronger performance. AffectGPT achieved 48.07% on MER-Caption+ and 52.17% on OV-MERD, which significantly outperformed general multimodal large language models. The Clue2Emo method further improved the results to 50.49% and 55.02%, respectively, achieving an absolute improvement of 2.4% on both benchmarks. It is worth noting that Clue2Emo achieved the best performance, which shows its superiority in fine-grained descriptive recognition. Comparisons were also made with smaller encoders (CLIP-ViTB / 32 and Chinese-HuBERT-Base), indicated by "-SE" in the table. While AffectGPT-SE and Clue2Emo-SE both had lower absolute scores compared to their larger encoder variants, Clue2Emo-SE consistently outperformed AffectGPT-SE, even surpassing the full AffectGPT model. These results demonstrate that even with reduced encoder capacity, the framework and training strategy can provide a sustained advantage.

[0094] Table 2. Ablation study results on OV-MERD and MERCaption+

[0095]

[0096] Table 2 shows that removing the second stage (integrated cue reasoning) causes the largest performance drop, reducing the average by over 15 on MER-Caption+ and over 7 on OV-MERD. This highlights the crucial role of the two-stage training strategy in leveraging descriptive supervision. Performance also drops when the second-stage cue excludes either video or audio. This drop is smaller than when both are removed simultaneously, suggesting that each modality provides complementary cues, and both are necessary for achieving good performance.

[0097] This embodiment also provides a computer device applicable to a brain-inspired open-vocabulary multimodal emotion recognition method, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the brain-inspired open-vocabulary multimodal emotion recognition method as proposed in the above embodiment.

[0098] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0099] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements a brain-inspired open-vocabulary multimodal emotion recognition method as proposed in the above embodiment.

[0100] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A brain-inspired open-vocabulary multimodal emotion recognition method, characterized in that, include: The audio, video, and text data of the object to be identified are acquired and feature extraction is performed to obtain acoustic features, visual features, and semantic features. The acoustic features, visual features, and semantic features are then mapped to a unified emotional semantic space to form multimodal features. Based on the audio and video data of the objects to be identified, an emotion cue dataset MER-CLUE is constructed; The step of constructing the emotional cue dataset MER-CLUE based on the audio and video data of the object to be identified includes: inputting the audio and video data of the object to be identified into a preset cue generation base big language model, generating audio emotional cue text and video emotional cue text related to the emotional state by the cue generation base big language model; collecting and storing the generated audio emotional cue text and video emotional cue text to form the emotional cue dataset MER-CLUE used to describe multimodal emotional sensory evidence; The LoRA method is used to adjust the parameters of a pre-defined cue generation base language model. This model is then optimized based on the MER-CLUE sentiment cue dataset to obtain a new cue generation language model. The optimization of the cue generation base language model based on the MER-CLUE sentiment cue dataset includes constructing audio and video sentiment cue descriptions corresponding to the cue generation base language model, formally expressed as follows: ； ； in, It is an acoustic emotion embedding representation. It is a visual emotion embedding representation. It is a semantic sentiment embedding representation. It is a semantic feature vector. It refers to the cue descriptions generated by the cue generation base large language model. LLM is a pre-defined cue generation base large language model. It is the cross-entropy loss that ensures the generated text matches the cues in MER-CLUE. Acoustic emotion embedding representation of the i-th sample It is the visual sentiment embedding representation of the i-th sample. It is the semantic sentiment embedding representation of the i-th sample. It is the semantic feature vector of the i-th sample. Let be the emotional cue text corresponding to the i-th sample, N be the total number of samples, and logP(|) be the log probability; After training, the resulting large language model with fine-grained cue generation capability is solidified into a large language model for cue generation. Based on the big language model for cue generation, audio cue text and video cue text are generated from the audio data and video data of the object to be identified, respectively, and then concatenated to form a cue prompt. The cue prompt and multimodal features are used as the inference input of the big language model for cue generation. Based on reasoning input containing clues, the LoRA method is used to train the pre-defined emotional reasoning foundation big language model for emotional reasoning training; Based on the sentiment reasoning big language model trained by sentiment reasoning, open-vocabulary sentiment descriptions are generated, and finally sentiment recognition results are output.

2. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 1, characterized in that, The process of mapping the acoustic features, visual features, and semantic features to a unified emotional semantic space to form multimodal features includes: The audio data x of the input object to be identified are respectively... a Video data x v and text data x t Feature extraction is performed, and acoustic feature vectors are obtained using an audio encoder. The visual feature vector h is obtained using a video encoder. ᵥ The semantic feature vector h is obtained using a text encoder. t ; acoustic feature vectors Mapping to the sentiment semantic space yields an acoustic sentiment embedding representation. ; the visual feature vector h ᵥ Mapping to the sentiment semantic space, we obtain the visual sentiment embedding representation z. ᵥ ; the text feature vector h t Mapping to the sentiment semantic space, we obtain the semantic sentiment embedding representation z. t ; Embedding the acoustic emotion representation Visual emotion embedding representation z ᵥ and semantic sentiment embedding representation z t Alignment and integration are carried out in a unified emotional semantic space.

3. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 1, characterized in that, The audio and video data of the object to be identified are used to generate audio cue text and video cue text respectively, and then concatenated to form a cue prompt. The cue prompt and multimodal features are used as the inference input of the cue generation large language model, including: Audio emotional cue text and video emotional cue text are generated using a large language model for cue generation, respectively. The audio emotional cue text describes acoustic emotional evidence such as pitch changes, energy fluctuations, and speech rate changes, while the video emotional cue text describes visual emotional evidence such as local facial expressions, posture dynamics, and movement amplitude. The audio emotional cue text and the video emotional cue text are concatenated in a preset order to construct a cue prompt containing cross-modal emotional evidence; The aforementioned clues and multimodal features are input into the clue generation language model, enabling the model to generate emotional description text in open-vocabulary form during the reasoning stage by simultaneously utilizing the fused sensory cue text and low-level multimodal evidence.

4. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 3, characterized in that, The generation of sentiment description text in open-vocabulary form includes: The formalization of the overall generation process is as follows: ; in, It is all the text in the inference text describing the true sentiment of the i-th sample. LLM is a pre-defined cue generation base large language model. It is an acoustic emotion embedding representation. It is a visual emotion embedding representation. It is a semantic sentiment embedding representation. is the semantic feature vector, and c is the context condition; The training objective is formalized as follows: ； in, It is a token or the entire sequence in the true sentiment description text of the i-th sample. Acoustic emotion embedding representation of the i-th sample It is the visual sentiment embedding representation of the i-th sample. It is the semantic sentiment embedding representation of the i-th sample. It is the semantic feature vector of the i-th sample. Here, is the context condition for the i-th sample, N is the total number of samples, and logP(|) is the log probability. It is the optimization goal.

5. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 1, characterized in that, Based on reasoning input containing clues, the LoRA method is used to train a pre-defined sentiment reasoning foundation large language model for sentiment reasoning. This includes: using the LoRA method to perform low-rank decomposition and inject trainable parameters into the weight matrix of the sentiment inference base language model, including the attention mechanism layer and / or feedforward network layer, and updating the newly added low-rank parameters with gradients; calculating the sentiment classification loss for the sentiment inference base language model based on the sentiment category of the sample labels, and outputting the sentiment classification loss to supervise the optimization of the LoRA parameters.

6. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 1, characterized in that, The sentiment reasoning big language model trained based on sentiment reasoning generates open-vocabulary sentiment descriptions and finally outputs sentiment recognition results, including: The emotional reasoning big language model is used to generate open-vocabulary emotional description text under the joint conditions of multimodal features and clue prompts. The emotional description text is used to characterize the fine-grained emotional state of the input sample in a natural language manner. The emotional description text is semantically parsed or classified and mapped to obtain the corresponding emotional recognition result. The emotional recognition result includes a preset emotional category, emotional intensity or emotional label.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the brain-inspired open-vocabulary multimodal emotion recognition method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the brain-inspired open-vocabulary multimodal emotion recognition method according to any one of claims 1 to 6.

Images