Brain-inspired open vocabulary multi-mode emotion recognition method

Abstract

The invention discloses a brain-inspired open vocabulary multi-modal emotion recognition method, and relates to the technical field of multi-modal emotion calculation, and the method comprises the steps: obtaining audio data, video data and text data of a to-be-recognized object, carrying out the feature extraction, obtaining an acoustic feature, a visual feature and a semantic feature, and carrying out the mapping to form a multi-modal feature; constructing an emotional clue data set MER-CLUE, and performing parameter adjustment on the clue generation base large language model by adopting a LoRA method to obtain a clue generation large language model; generating a clue prompt by using a clue generation large language model; and performing emotion reasoning training on the emotion reasoning base large language model, generating open vocabulary emotion description based on the emotion reasoning base large language model, and outputting an emotion recognition result. According to the method, multi-modal unified emotion expression is realized, and explainable open vocabulary emotion description is generated; and in combination with optimization, emotion recognition accuracy, generalization and training efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal affective computing, and in particular to a brain-inspired open-vocabulary multi-modal emotion recognition method. BACKGROUND

[0002] With the development of artificial intelligence technology, affective computing has gradually evolved from single-modal emotion recognition to multi-modal affective understanding. Existing emotion recognition methods usually rely on a pre-set emotion label system, and classify features of a single modality such as audio, image or text, which is difficult to accurately depict the multi-dimension, fine-grained and context-dependent nature of human emotions. In addition, traditional models have limited ability to model the causes of emotions, changes in emotional states and the associated information between different modalities in complex scenarios, resulting in weak generalization and insufficient explainability of emotion recognition results.

[0003] In recent years, large language models (LLM) have shown strong semantic reasoning capabilities in natural language understanding and generation tasks. However, there are still many challenges in directly applying large language models to multi-modal emotion recognition: first, audio features, visual features and semantic features are difficult to represent in a unified space, and the model cannot fully capture the cross-modal associations between different modalities; second, the traditional classification-based emotion label system limits the model's expression range, making it impossible for the model to output open-vocabulary, natural language form emotion explanations; third, there is a lack of effective cross-modal emotion cue supervision mechanism, making it difficult for large language models to learn fine-grained understanding of human emotions, emotion performance and emotion details.

[0004] In addition, the complete training of large language models is costly and has a huge number of parameters, which is not conducive to efficient and customized optimization in specific emotion recognition tasks. Although parameter-efficient fine-tuning techniques such as LoRA can reduce training costs, how to combine multi-modal emotion supervision signals to enable large language models to have both explainable emotion description generation capabilities and stable emotion classification capabilities remains a key problem that existing technology has not effectively solved.

[0005] Therefore, there is an urgent need for a multi-modal emotion reasoning method that can integrate multi-modal features in a unified emotion semantic space and generate and supervise emotion cues to enable large language models to have fine-grained emotion reasoning capabilities and output open-vocabulary emotion descriptions and emotion recognition results. SUMMARY

[0006] The present application is proposed in view of the above-mentioned problems.

[0007] Therefore, the problem to be solved by the present application is how to endow the computing model with explicit reasoning ability similar to the human brain in the open-vocabulary multi-modal sentiment recognition task, so that it can generate accurate, fine-grained, and interpretable and traceable recognition results in the decision-making process on the sentiment description for the given audio, video and text input, while significantly improving the accuracy of open-vocabulary sentiment recognition.

[0008] To solve the above technical problems, the present application provides the following technical solutions:

[0009] In a first aspect, the present application provides a brain-inspired open-vocabulary multi-modal emotion recognition method, comprising: obtaining audio data, video data and text data of an object to be recognized and performing feature extraction to obtain acoustic features, visual features and semantic features; mapping the acoustic features, visual features and semantic features to a unified sentiment semantic space to form multi-modal features;

[0010] On the basis of the audio data and video data of the object to be recognized, a sentiment clue data set MER-CLUE is constructed;

[0011] The preset clue generation base large language model is adjusted in parameters by using the LoRA method, and the clue generation base large language model is optimized based on the sentiment clue data set MER-CLUE to obtain a clue generation large language model;

[0012] Based on the clue generation large language model, audio clue text and video clue text are respectively generated for the audio data and video data of the object to be recognized, and are spliced to form a clue prompt, and the clue prompt and the multi-modal features are taken as the reasoning input of the clue generation large language model;

[0013] Based on the reasoning input containing the clue prompt, the sentiment reasoning base large language model is trained in sentiment reasoning by using the LoRA method;

[0014] Based on the sentiment reasoning large language model obtained through the sentiment reasoning training, an open-vocabulary emotion description is generated, and finally a sentiment recognition result is output.

[0015] As a preferred scheme of the brain-inspired open-vocabulary multi-modal emotion recognition method, the acoustic features, visual features and semantic features are mapped to a unified sentiment semantic space to form multi-modal features, which comprises:

[0016] The input audio data x a , video data x v and text data x t of the object to be recognized are respectively processed by feature extraction, the acoustic feature vector is obtained by using an audio encoder, and the visual feature vector obtaining a semantic feature vector by using a text encoder ;

[0017] mapping the acoustic feature vector to a sentiment semantic space to obtain an acoustic sentiment embedding representation ; mapping the visual feature vector to a sentiment semantic space to obtain a visual sentiment embedding representation ; mapping the text feature vector to a sentiment semantic space to obtain a semantic sentiment embedding representation ;

[0018] aligning and fusing the acoustic sentiment embedding representation , the visual sentiment embedding representation and the semantic sentiment embedding representation in a unified sentiment semantic space.

[0019] As a preferred scheme of the brain-inspired open-vocabulary multi-modal emotion recognition method, the audio data and the video data of the object to be recognized are used to construct a sentiment clue data set MER-CLUE, which includes:

[0020] The audio data and the video data of the object to be recognized are input into a preset clue generation base large language model, and the clue generation base large language model generates audio sentiment clue text and video sentiment clue text related to the emotional state; the generated audio sentiment clue text and video sentiment clue text are collected and stored to form a sentiment clue data set MER-CLUE for describing multi-modal sentiment sensory evidence.

[0021] As a preferred scheme of the brain-inspired open-vocabulary multi-modal emotion recognition method, the clue generation base large language model is optimized based on the sentiment clue data set MER-CLUE to obtain a clue generation large language model, which includes:

[0022] The audio emotional clue description and the video emotional clue description corresponding to the clue generation base large language model are constructed, and the formalized expression is as follows:

[0023] ;

[0024] ;

[0025] wherein, is the acoustic sentiment embedding representation, is the visual sentiment embedding representation, is the semantic sentiment embedding representation, is the semantic feature vector, is a clue description generated by a clue generation base large language model, and the LLM is a preset clue generation base large language model, is a cross-entropy loss for ensuring that the generated text matches the clue in the MER-CLUE, is an acoustic emotional embedding representation of the i-th sample, is a visual emotional embedding representation of the i-th sample, is a semantic emotional embedding representation of the i-th sample, is a semantic feature vector of the i-th sample, is emotional clue text corresponding to the i-th sample, N is the total number of samples, and logP(|) is a logarithmic probability.

[0026] After the training is completed, the obtained clue generation base large language model with fine-grained clue generation capability is solidified as a clue generation large language model.

[0027] As a preferred scheme of the brain-inspired open-vocabulary multi-modal emotion recognition method, wherein: audio data and video data of the object to be recognized are respectively generated to form audio clue text and video clue text, and the audio clue text and the video clue text are spliced to form a clue prompt, and the clue prompt and multi-modal features are input as inference input of the clue generation large language model, comprising:

[0028] The audio emotional clue text and the video emotional clue text are respectively generated by using the clue generation large language model, the audio emotional clue text describes acoustic emotional evidence of pitch variation, energy fluctuation and speech rate variation, and the video emotional clue text describes visual emotional evidence of local facial expression, posture dynamics and action amplitude;

[0029] The audio emotional clue text and the video emotional clue text are spliced in a preset order to construct a clue prompt containing cross-modal emotional evidence;

[0030] The clue prompt and the multi-modal features are input into the clue generation large language model, so that the clue generation large language model simultaneously uses the fused sensory clue text and low-level multi-modal evidence in the inference stage to generate an open-vocabulary emotional description text.

[0031] As a preferred scheme of the brain-inspired open-vocabulary multi-modal emotion recognition method, wherein: the open-vocabulary emotional description text is generated, comprising: The formalization of the overall generation process is as follows: ;

[0032] Wherein, is all the text in the true emotion description inference text of the ith sample, LLM is a preset clue generation base large language model, is an acoustic sentiment embedding representation, is a visual sentiment embedding representation, is a semantic sentiment embedding representation, is a semantic feature vector, and c is a context condition.

[0033] The formalization of the training target is as follows:

[0034]

[0035] wherein, is a certain token or the entire sequence in the true emotion description text of the ith sample, is an acoustic sentiment embedding representation of the ith sample, is a visual sentiment embedding representation of the ith sample, is a semantic sentiment embedding representation of the ith sample, is a semantic feature vector of the ith sample, is a context condition of the ith sample, N is the total number of samples, and logP(|) is a logarithmic probability, is an optimization target.

[0036] As a preferred scheme of the brain-inspired open-vocabulary multi-modal emotion recognition method, wherein: on the basis of the inference input containing the clue prompt, the LoRA method is used to perform emotion inference training on the emotion inference base large language model, including: using the LoRA method to perform low-rank decomposition and trainable parameter injection on the weight matrix in the attention mechanism layer and / or the feedforward network layer of the emotion inference base large language model, performing gradient update on the newly added low-rank parameters; calculating the emotion inference base large language model according to the sample labeled emotion category, and outputting an emotion classification loss, and using the emotion classification loss to supervise the optimization of the LoRA parameters.

[0037] As a preferred scheme of the brain-inspired open-vocabulary multi-modal emotion recognition method, wherein: based on the emotion inference large language model obtained through emotion inference training, an open-vocabulary emotion description is generated, and finally an emotion recognition result is output, including:

[0038] The emotion inference large language model is used to generate an open-vocabulary emotion description text under the joint condition of multi-modal features and clue prompts, and the emotion description text is used to describe the fine-grained emotion state of the input sample in a natural language manner; the emotion description text is subjected to semantic analysis or classification mapping to obtain a corresponding emotion recognition result, and the emotion recognition result includes a preset emotion category, an emotion intensity or an emotion label. ​

[0039] In a second aspect, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, they implement the steps of a brain-inspired open-vocabulary multimodal emotion recognition method as described in the first aspect of the present invention.

[0040] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, they implement the steps of a brain-inspired open-vocabulary multimodal emotion recognition method as described in the first aspect of the present invention.

[0041] The beneficial effects of this invention are as follows: By introducing the sentiment cue generation and sentiment inference mechanism of a large language model, this invention achieves fine-grained understanding of multimodal emotional states. The large language model is optimized based on MER-CLUE-based supervised signals, enabling the model to generate interpretable open-vocabulary sentiment descriptions, significantly improving the expressive and generalization abilities of sentiment recognition. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0047] Example 1

[0048] Referring to Figure 1 For the first embodiment of the present application, the embodiment provides a brain-inspired open-vocabulary multi-modal emotion recognition method, comprising:

[0049] S101: Acquire audio data, video data and text data of the to-be-recognized object and perform feature extraction to obtain acoustic features, visual features and semantic features, map the acoustic features, visual features and semantic features to a unified emotional semantic space to form multi-modal features.

[0050] Further, the input audio data x a , video data x v and text data x t of the to-be-recognized object are respectively subjected to feature extraction processing, acoustic feature vectors are obtained by using an audio encoder, visual feature vectors are obtained by using a video encoder, and semantic feature vectors are obtained by using a text encoder;

[0051] The acoustic feature vectors are mapped to an emotional semantic space to obtain acoustic emotional embedding representations ; the visual feature vectors are mapped to the emotional semantic space to obtain visual emotional embedding representations ; and the text feature vectors are mapped to the emotional semantic space to obtain semantic emotional embedding representations ;

[0052] The acoustic emotional embedding representations , visual emotional embedding representations and semantic emotional embedding representations are aligned and fused in the unified emotional semantic space.

[0053] S102: Based on the audio data and video data of the to-be-recognized object, a sentiment clue dataset MER-CLUE is constructed.

[0054] Further, the audio data and video data of the to-be-recognized object are input into a preset clue generation base large language model, and the clue generation base large language model generates audio emotional clue texts and video emotional clue texts related to emotional states; the generated audio emotional clue texts and video emotional clue texts are collected and stored to form a sentiment clue dataset MER-CLUE for describing multi-modal emotional sensory evidence.

[0055] It should be noted that in the video part, first, the Qwen2.5-VL-7B-Instruct model is used to analyze the video under the preset prompt constraint, to generate text clues describing emotion-related subtle signals, including but not limited to facial expression changes, eye movement features, body posture, voice tone changes, and environmental context, etc. The generated clue text does not directly give the emotion category judgment, but serves as an intermediate semantic representation in multi-modal emotion understanding. The video clue text and the audio clue text are sorted and labeled to form the emotion clue dataset MER-CLUE. This dataset is used as the first stage training data for the clue generation base large language model, so that the clue generation base large language model has the ability to generate fine-grained emotion clue descriptions from multi-modal features.

[0056] In the audio part, the Kimi-Audio-7B-Instruct model is used to generate structured emotion clues through prompts to guide the model to analyze speech features from the perspectives of phonetics and psychology. The prompts are used to constrain the model to focus on acoustic indicators such as pitch, volume, speed, clarity, breathing pattern, vocal cord tension, and pause, and guide the model to establish the relationship between the acoustic indicators and emotion state, emotion intensity, and valence, thereby generating acoustic emotion clue text for emotion understanding. The clue text does not directly give the emotion category judgment, but serves as an intermediate semantic representation in the multi-modal emotion reasoning process.

[0057] The emotion clue dataset MER-CLUE contains 28194 multi-modal samples with structured annotations, including primary and secondary emotions, facial, body, and acoustic clues, as well as quantified intensity levels. It has a wide coverage, with an average of 730 words per sample and more than 35000 unique emotion words. This scale and richness make the emotion clue dataset MER-CLUE a high-quality resource for interpretable OV-MER research.

[0058] S103: Adjust the parameters of the preset clue generation base large language model using the LoRA method, and optimize the clue generation base large language model based on the emotion clue dataset MER-CLUE to obtain a clue generation large language model.

[0059] Further, the audio emotion clue description and the video emotion clue description corresponding to the clue generation base large language model are constructed, which are formally expressed as follows:

[0060] ;

[0061] ;

[0062] wherein, is an acoustic sentiment embedding representation, is a visual sentiment embedding representation, is a semantic sentiment embedding representation, is a semantic feature vector, is a clue description generated by a clue generation base large language model, LLM is a preset clue generation base large language model, is a cross-entropy loss that ensures that the generated text matches the clue in the MER-CLUE, is an acoustic sentiment embedding representation of the i-th sample, is a visual sentiment embedding representation of the i-th sample, is a semantic sentiment embedding representation of the i-th sample, is a semantic feature vector of the i-th sample, is an emotional clue text corresponding to the i-th sample, N is the total number of samples, and logP(|) is a logarithmic probability;

[0063] After the training is completed, the obtained clue generation base large language model with fine-grained clue generation capability is solidified as a clue generation large language model.

[0064] It should be noted that during the optimization process, the audio, video and fusion projector parameters remain trainable, while the large language model is subjected to LoRA parameter efficient fine-tuning strategy to enhance the model's generation capability from sensory features to clue descriptions. After a preset number of training rounds, the obtained model is referred to as an emotional clue generation model, which can generate fine-grained audio clue text and video clue text according to multi-modal input. The emotional clue generation model is used in the subsequent clue prompt-based open vocabulary sentiment reasoning training stage (i.e., stage two), providing interpretable intermediate clue representation for this stage, thereby simulating the temporary storage and encoding function of human working memory in the emotional cognitive process.

[0065] S104: Based on the clue generation large language model, audio clue text and video clue text are generated for the audio data and video data of the object to be recognized, respectively, and are spliced to form a clue prompt, which is taken as inference input of the clue generation large language model together with the multi-modal features.

[0066] Further, audio emotional clue text and video emotional clue text are respectively generated by the clue generation large language model, the audio emotional clue text describes acoustic emotional evidence such as pitch change, energy fluctuation and speech rate change, and the video emotional clue text describes visual emotional evidence such as facial local expression, posture dynamics and action amplitude;

[0067] The audio emotional clue text and the video emotional clue text are spliced in a preset order to construct a clue prompt containing cross-modal emotional evidence;

[0068] The clue prompt is input into a clue generation large language model together with the multi-modal features, so that the clue generation large language model simultaneously utilizes the fused sensory clue text and low-level multi-modal evidence in an inference stage to generate emotion description text in an open vocabulary form.

[0069] Further, the formalization of the overall generation process is as follows: ;

[0070] wherein, is all text in the true emotion description inference text of the i-th sample, LLM is a preset clue generation base large language model, is an acoustic sentiment embedding representation, is a visual sentiment embedding representation, is a semantic sentiment embedding representation, is a semantic feature vector, and c is a context condition;

[0071] The formalization of the training target is as follows:

[0072] ;

[0073] wherein, is a certain token or the entire sequence in the true emotion description text of the i-th sample, is an acoustic sentiment embedding representation of the i-th sample, is a visual sentiment embedding representation of the i-th sample, is a semantic sentiment embedding representation of the i-th sample, is a semantic feature vector of the i-th sample, is a context condition of the i-th sample, N is the total number of samples, and logP(|) is a logarithmic probability, is an optimization target.

[0074] S105: Based on the inference input containing the clue prompt, a LoRA method is used to perform emotion inference training on the emotion inference base large language model.

[0075] Further, the LoRA method is used to perform low-rank decomposition and trainable parameter injection on the weight matrix in the attention mechanism layer and / or the feedforward network layer of the emotion inference base large language model, the newly added low-rank parameters are updated by gradients, the emotion inference base large language model is calculated according to the sample labeled emotion category, and an emotion classification loss is outputted, so as to supervise the optimization of the LoRA parameters by the emotion classification loss.

[0076] It should be noted that after the multi-modal features and the clue prompt vector are fused, the fused multi-modal input sequence is input into the sentiment reasoning base large language model for supervised fine-tuning: low-rank decomposition is performed, and trainable low-rank parameters are introduced; a training sample with an emotion label is constructed, and each training sample includes: image region features, scene features, fine-grained clue features; text description or dialogue content; clue prompt vector; target emotion label;

[0077] The sentiment reasoning base large language model generates a predicted emotion distribution after receiving the multi-modal input sequence, and uses cross-entropy loss. In backpropagation, only the LoRA low-rank matrix is updated, and the remaining sentiment reasoning base large language model parameters remain frozen; the updated LoRA parameters gradually learn the influence mechanism of image clues and text clues on emotions, thereby significantly improving the model reasoning ability.

[0078] After training, a sentiment reasoning large language model is obtained. In the reasoning phase, the multi-modal features corresponding to the object to be recognized and the clue prompts output by the sentiment clue generation model are jointly used as model inputs. The sentiment reasoning large language model automatically outputs an open-vocabulary emotion description text based on its learned multi-modal sentiment reasoning ability without human intervention through a self-recurrent text generation method; and further analyzes the emotion state according to the emotion semantics contained in the emotion description text to obtain the corresponding emotion classification result.

[0079] S106: Based on the sentiment reasoning large language model obtained by sentiment reasoning training, an open-vocabulary emotion description is generated, and a sentiment recognition result is finally output.

[0080] Further, the sentiment reasoning large language model is used to generate an open-vocabulary emotion description text under the joint condition of multi-modal features and clue prompts. The emotion description text is used to describe the fine-grained emotion state of the input sample in a natural language manner; the emotion description text is semantically analyzed or classified mapped to obtain a corresponding emotion recognition result, which includes a preset emotion category, emotion intensity, or emotion label.

[0081] It should be noted that in this training phase, the MER-Caption+ dataset is used as a training sample, and the clue prompt and multi-modal feature are introduced to train the sentiment reasoning large language model for sentiment reasoning, so that the sentiment reasoning large language model learns to perform comprehensive reasoning based on the sentiment clue and outputs an open-vocabulary emotion description and an emotion recognition result.

[0082] Embodiment 2

[0083] The embodiment is a second embodiment of the application, and provides a brain-inspired open-vocabulary multi-modal emotion recognition method. In order to verify the beneficial effects of the application, a simulation experiment is performed for scientific demonstration.

[0084] The experimental setup follows the MER-FG track requirements of MER2025, with adaptive adjustments in dataset division. Specifically, in the MER-Caption+ dataset, 90% of the samples (28194) are selected as the training set, and the remaining 10% of the samples (3133) are selected as the test set. In addition, the OV-MERD dataset is introduced as an independent test set to evaluate the cross-dataset generalization ability of the model in the open-vocabulary emotion recognition task. Finally, the performance of the model is evaluated on the MER-Caption+ test set and the OV-MERD test set, respectively. Since the label space is open and multi-labeled, the predicted results and the true labels are first standardized by lemmatization and synonym merging . Based on the Plutchik emotion wheel, the emotion words are mapped to the coarse-grained layer and the fine-grained layer . In order to reduce the bias caused by single division, a set of five emotion wheels is used. For each emotion wheel and layer , the precision, recall and F1 value of the sample level are calculated, denoted as . The average value of the final coarse and fine-grained scores is calculated as follows:

[0085] ;

[0086] wherein, is the comprehensive evaluation index on the kth emotion granularity level, N is the total number of samples, j is the sample index, and k is the emotion granularity level index.

[0087] Table 1. Comparison of S1 and S2 scores on OV-MERD and MER-Caption+

[0088]

[0089] wherein, S1 is the macro-averaged S1 evaluation index of the model on the coarse-grained emotion level, S2 is the macro-averaged S2 evaluation index of the model on the fine-grained emotion level, and avg is the average value of S1 and S2.

[0090] Table 1 shows the results on MER-Caption+ and OV-MERD. API-based models (ChatGPT-4o, Gemini2.5Pro) score the lowest, with an average of less than 41% on MER-Caption+. This is mainly because they cannot handle the audio modality and can only rely on text and visual information. In contrast, open-source multi-modal models (Phi-4-MM

[18] , Qwen2.5-Omni-7B

[19] ) can integrate video and audio cues, which allows them to achieve significant improvements on both datasets. Methods specific to the affective domain show stronger performance. AffectGPT achieves 48.07% on MER-Caption+ and 52.17% on OV-MERD, already outperforming general-purpose multi-modal large language models. The method Clue2Emo further improves the results to 50.49% and 55.02%, achieving an absolute improvement of 2.4% on both benchmarks. It is worth noting that Clue2Emo achieves the best performance, which indicates its superiority in fine-grained descriptive recognition. Comparisons are also made on smaller encoders (CLIP-ViTB / 32 and Chinese-HuBERT-Base), denoted by “-SE” in the table. Although AffectGPT-SE and Clue2Emo-SE have lower absolute scores compared to their large encoder variants, Clue2Emo-SE consistently outperforms AffectGPT-SE, even surpassing the full AffectGPT model. These results show that even with reduced encoder capacity, the framework and training strategy can bring sustained advantages.

[0091] Table 2. Ablation study results on OV-MERD and MERCaption+

[0092]

[0093] Table 2 shows that removing the second stage (comprehensive cue reasoning) results in the largest performance drop, reducing the average by more than 15 on MER-Caption+ and more than 7 on OV-MERD. This highlights the key role of the two-stage training strategy in leveraging descriptive supervision. When the second-stage prompt excludes either video or audio, performance also decreases. This decrease is smaller than when both are removed, indicating that each modality provides complementary cues and that both are necessary to achieve good performance.

[0094] The embodiment also provides a computer device suitable for the case of the brain-inspired open-vocabulary multi-modal emotion recognition method, and the computer device comprises a memory and a processor.

[0095] The computer device can be a terminal, and the computer device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0096] The embodiment also provides a storage medium having a computer program stored thereon, and the program is executed by a processor to implement the brain-inspired open-vocabulary multi-modal emotion recognition method.

[0097] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and all modifications or replacements should be included in the scope of the claims of the present application.

Claims

1. A brain-inspired open-vocabulary multimodal emotion recognition method, characterized in that, include: The audio, video, and text data of the object to be identified are acquired and feature extraction is performed to obtain acoustic features, visual features, and semantic features. The acoustic features, visual features, and semantic features are then mapped to a unified emotional semantic space to form multimodal features. Based on the audio and video data of the objects to be identified, an emotion cue dataset MER-CLUE is constructed; The LoRA method is used to adjust the parameters of the preset cue generation base large language model, and the cue generation base large language model is optimized based on the sentiment cue dataset MER-CLUE to obtain the cue generation large language model. Based on the big language model for cue generation, audio cue text and video cue text are generated from the audio data and video data of the object to be identified, respectively, and then concatenated to form a cue prompt. The cue prompt and multimodal features are used as the inference input of the big language model for cue generation. Based on reasoning input containing clues, the LoRA method is used to train the pre-defined emotional reasoning foundation large language model for emotional reasoning training; Based on the sentiment reasoning big language model trained by sentiment reasoning, open-vocabulary sentiment descriptions are generated, and finally sentiment recognition results are output.

2. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 1, characterized in that, The process of mapping the acoustic features, visual features, and semantic features to a unified emotional semantic space to form multimodal features includes: The audio data x of the input object to be identified are respectively... a Video data x v and text data x t Feature extraction is performed, and acoustic feature vectors are obtained using an audio encoder. Visual feature vectors are obtained using a video encoder. Semantic feature vectors are obtained using a text encoder. ; acoustic feature vectors Mapping to the sentiment semantic space yields an acoustic sentiment embedding representation. ; Visual feature vector Mapping to the sentiment semantic space yields a visual sentiment embedding representation. ; to transform text feature vectors Mapping to the sentiment semantic space yields a semantic sentiment embedding representation. ; Embedding the acoustic emotion representation Visual emotion embedding representation and semantic sentiment embedding representation Alignment and integration are carried out in a unified emotional semantic space.

3. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 1, characterized in that, The step involves constructing the MER-CLUE emotional cue dataset based on the audio and video data of the object to be identified, including: The audio and video data of the object to be identified are input into a preset cue generation base big language model, which generates audio emotional cue text and video emotional cue text related to the emotional state. The generated audio emotional cue text and video emotional cue text are collected and stored to form the emotional cue dataset MER-CLUE, which is used to describe multimodal emotional sensory evidence.

4. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 2, characterized in that, The optimization of the cue generation base language model based on the sentiment cue dataset MER-CLUE to obtain a cue generation language model includes: The formal descriptions of audio and video emotion cues corresponding to the large language model for cue generation are as follows: ； ； in, It is an acoustic emotion embedding representation. It is a visual emotion embedding representation. It is a semantic sentiment embedding representation. It is a semantic feature vector. It refers to the cue descriptions generated by the cue generation base large language model. LLM is a pre-defined cue generation base large language model. It is the cross-entropy loss that ensures the generated text matches the cues in MER-CLUE. It is the acoustic emotion embedding representation of the i-th sample. It is the visual sentiment embedding representation of the i-th sample. It is the semantic sentiment embedding representation of the i-th sample. It is the semantic feature vector of the i-th sample. Let be the emotional cue text corresponding to the i-th sample, N be the total number of samples, and logP(|) be the log probability; After training, the resulting large language model with fine-grained cue generation capabilities is solidified into a large language model for cue generation.

5. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 2, characterized in that, The audio and video data of the object to be identified are used to generate audio cue text and video cue text respectively, and then concatenated to form a cue prompt. The cue prompt and multimodal features are used as the inference input of the cue generation large language model, including: Audio emotional cue text and video emotional cue text are generated using a large language model for cue generation, respectively. The audio emotional cue text describes acoustic emotional evidence such as pitch changes, energy fluctuations, and speech rate changes, while the video emotional cue text describes visual emotional evidence such as local facial expressions, posture dynamics, and movement amplitude. The audio emotional cue text and the video emotional cue text are concatenated in a preset order to construct a cue prompt containing cross-modal emotional evidence; The aforementioned clues and multimodal features are input into the clue generation language model, enabling the model to generate emotional description text in open-vocabulary form during the reasoning stage by simultaneously utilizing the fused sensory cue text and low-level multimodal evidence.

6. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 5, characterized in that, The generation of sentiment description text in open-vocabulary form includes: The overall generation process is formalized as follows: ； in, It is all the text in the inference text describing the true sentiment of the i-th sample. LLM is a pre-defined cue generation base large language model. It is an acoustic emotion embedding representation. It is a visual emotion embedding representation. It is a semantic sentiment embedding representation. is the semantic feature vector, and c is the context condition; The training objective is formalized as follows: ； in, It is a token or the entire sequence in the true sentiment description text of the i-th sample. It is the acoustic emotion embedding representation of the i-th sample. It is the visual sentiment embedding representation of the i-th sample. It is the semantic sentiment embedding representation of the i-th sample. It is the semantic feature vector of the i-th sample. Here, is the context condition for the i-th sample, N is the total number of samples, and logP(|) is the log probability. It is the optimization goal.

7. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 1, characterized in that, The step of training the sentiment reasoning base large language model using the LoRA method on the inference input containing clues includes: using the LoRA method to perform low-rank decomposition and inject trainable parameters into the weight matrix of the sentiment reasoning base large language model, including the attention mechanism layer and / or feedforward network layer, and updating the newly added low-rank parameters with gradients; calculating the sentiment classification loss for the sentiment reasoning base large language model according to the sentiment category labeled in the samples, and outputting the sentiment classification loss to supervise the optimization of the LoRA parameters.

8. The brain-inspired open-vocabulary multimodal emotion recognition method as described in claim 1, characterized in that, The sentiment reasoning big language model trained based on sentiment reasoning generates open-vocabulary sentiment descriptions and finally outputs sentiment recognition results, including: The emotional reasoning big language model is used to generate open-vocabulary emotional description text under the joint conditions of multimodal features and clue prompts. The emotional description text is used to characterize the fine-grained emotional state of the input sample in a natural language manner. The emotional description text is semantically parsed or classified and mapped to obtain the corresponding emotional recognition result. The emotional recognition result includes a preset emotional category, emotional intensity or emotional label.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the brain-inspired open-vocabulary multimodal emotion recognition method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the brain-inspired open-vocabulary multimodal emotion recognition method according to any one of claims 1 to 8.

Images