An emotion recognition method and system based on multi-modal feature retrieval, a terminal and a storage medium

Abstract

The application relates to the technical field of data processing, and discloses an emotion recognition method, system, terminal and storage medium based on multi-modal feature retrieval, which comprises the following steps: collecting video signals and audio signals of a subject, converting the audio signals into text signals, and extracting video features, audio features and text features; performing retrieval on the video features, the audio features and the text features in a single-modal feature retrieval library to obtain enhanced features; aligning the enhanced features to a unified feature space through a mapping module, inputting the enhanced features into a double-branch structure, dynamically adjusting the weights of the enhanced features through a modal weight distributor, obtaining weighted enhanced features, performing query on the enhanced features in a multi-modal feature retrieval library, and obtaining multi-modal retrieval features; and inputting the weighted enhanced features and the multi-modal retrieval features into a multi-modal large model after fusion for processing, and outputting emotion recognition results of the subject. The application realizes accurate perception of individual emotional states.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an emotion recognition method, system, terminal, and computer-readable storage medium based on multimodal feature retrieval. Background Technology

[0002] Emotions are subjective human cognitive experiences that profoundly influence thinking and behavior. Emotion recognition technology aims to infer an individual's emotional state through multimodal information such as vision, text, and speech, and is widely applied in fields such as medical monitoring, nursing management, human-computer interaction, and autonomous driving. In medical and nursing scenarios, emotion recognition can assist medical staff in monitoring patients' emotional fluctuations in real time, especially for children with limited cognitive or expressive abilities and patients with chronic diseases, providing objective and continuous assessments of their emotional states and aiding in the early identification and intervention of symptoms such as fatigue, anxiety, or pain. In human-computer interaction scenarios such as autonomous driving, by perceiving the driver's emotions in real time, the system can adjust the environment and interaction strategies accordingly, improving safety and user experience.

[0003] Existing emotion recognition methods are mainly divided into two categories: unimodal methods and multimodal methods. Unimodal methods extract features using only a single information from a visual, text, or speech modality. They are greatly affected by occlusion, individual differences, and environmental interference, making it difficult to comprehensively capture emotional features, and their interpretability is limited. Multimodal methods improve recognition accuracy and robustness by fusing multiple types of information, but temporal, spatial, and semantic differences exist between different modalities, making feature fusion difficult and easily introducing redundant information. At the same time, individual differences and the similarity between different emotions also increase the complexity of recognition.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide an emotion recognition method, system, terminal, and computer-readable storage medium based on multimodal feature retrieval. This invention aims to address the problem in existing technologies for multimodal emotion recognition where data from different modalities often exhibit quality differences and inconsistencies, and where temporal asynchrony, missing data, or differences in emotional intensity may exist between modalities, leading to conflicts during cross-modal feature fusion and resulting in the inability to effectively recognize the emotions of the subjects.

[0006] To achieve the above objectives, the present invention provides an emotion recognition method based on multimodal feature retrieval, which includes the following steps:

[0007] Video and audio signals of the subjects are collected, the audio signals are converted into text signals, and the video signals, audio signals and text signals are respectively input into the corresponding feature extraction modules to obtain video features, audio features and text features;

[0008] Based on the video features, audio features, and text features, a search is performed in a pre-built single-modal feature retrieval library to obtain enhanced video features, enhanced audio features, and enhanced text features;

[0009] The enhanced video features, enhanced audio features, and enhanced text features are aligned to a unified feature space via a mapping module and input into a dual-branch structure. The weights of the enhanced video features, enhanced audio features, and enhanced text features are dynamically adjusted by a modal weight allocator to obtain weighted enhanced features. Based on the enhanced video features, enhanced audio features, and enhanced text features, a query is performed in a pre-built multimodal feature retrieval library to obtain multimodal retrieval features.

[0010] The weighted enhanced features are fused with the multimodal retrieval features and then input into a multimodal large model for processing, outputting the emotion recognition results of the subject.

[0011] Optionally, the emotion recognition method based on multimodal feature retrieval, wherein the step of retrieving enhanced video features, enhanced audio features, and enhanced text features from a pre-built single-modal feature retrieval library based on the video features, the audio features, and the text features, further includes:

[0012] Features are extracted in advance using existing public datasets and corresponding encoders, and single-modal feature retrieval libraries and multimodal feature retrieval libraries are constructed based on the extracted features.

[0013] Optionally, in the emotion recognition method based on multimodal feature retrieval, the step of inputting the video signal, the audio signal, and the text signal into corresponding feature extraction modules to obtain video features, audio features, and text features specifically involves:

[0014] video signal Audio signals With text signals Input the video encoder, audio encoder, and text encoder respectively to obtain video features. Audio features Text features .

[0015] Optionally, the emotion recognition method based on multimodal feature retrieval, wherein the step of retrieving enhanced video features, enhanced audio features, and enhanced text features from a pre-built single-modal feature retrieval database based on the video features, the audio features, and the text features to obtain enhanced video features, enhanced audio features, and enhanced text features specifically includes:

[0016] Cosine similarity is calculated and performed in dedicated search databases for each modality.K -Nearest neighbor search, for each input sample Retrieve the most similar ones K candidate samples , and Marked as intra-modal retrieval features ;

[0017] Original features and intramodal retrieval features Perform linear mapping to generate positive cross-attention queries respectively. Positive cross attention key and positive cross attention value Calculate the positive cross-attention weights between the original features and the in-modal retrieval features. :

[0018] ;

[0019] in, This is the scaling factor. For the Softmax function, Indicates transpose;

[0020] Calculate the product of the positive cross-attention value and the positive cross-attention weight to obtain the enhanced features after positive cross-attention retrieval. :

[0021] ;

[0022] In-modal retrieval features and original features Perform linear mapping to generate reverse cross-attention queries respectively. Reverse cross attention key and reverse cross attention value Calculate the inverse cross-attention weights between the retrieved features and the original features within the modality. :

[0023] ;

[0024] Calculate the product of the back cross-attention value and the back cross-attention weight to obtain the features enhanced by back cross-attention retrieval. :

[0025] ;

[0026] Enhanced features retrieved via positive cross-attention Enhanced features retrieved via reverse cross-attention The features are added together, and the residuals are then concatenated with the original features to create a single-modal augmentation feature. Among them, single-mode enhancement features Includes enhanced video features Enhance audio features Enhanced text features ;

[0027] Enhance features for unimodal use of classification loss. Constraints are imposed to ensure single-modal enhancement features. Validity:

[0028] ;

[0029] in, For unimodal emotion classification loss, For visual modality emotion classification loss, For audio modality emotion classification loss, The loss is used for text modality sentiment classification.

[0030] Optionally, the emotion recognition method based on multimodal feature retrieval, wherein aligning the enhanced video features, enhanced audio features, and enhanced text features to a unified feature space via a mapping module and inputting them into a dual-branch structure, and dynamically adjusting the weights of the enhanced video features, enhanced audio features, and enhanced text features through a modal weight allocator to obtain weighted enhanced features, specifically includes:

[0031] Enhanced video features Enhance audio features Enhanced text features Feature mapping is performed, and the data is projected onto a unified feature space through a multilayer perceptron to obtain aligned and enhanced video features. Alignment enhances audio features Enhanced text features with alignment This will align and enhance video features. Alignment enhances audio features Enhanced text features with alignment Input to the modal weight assigner;

[0032] The modality weight allocator dynamically calculates the modality importance weight distribution for each sample through a routing network consisting of a linear layer, batch normalization, and activation functions. , Represents video modal weights. Represents audio modal weights. Representing text modality weights, this method aligns and enhances video features based on the modality importance weight distribution for each sample. Alignment enhances audio features Enhanced text features with alignment After correction, the weighted enhanced features are obtained:

[0033] ;

[0034] ;

[0035] ;

[0036] The weighted enhancement features include weighted alignment enhancement video features. Weighted alignment enhances audio features Enhanced text features with weighted alignment ;

[0037] Constructing Modal Weight Assignment Loss :

[0038] ;

[0039] in, Representing modes The assigned weights, including , and ;

[0040] The step of querying a pre-built multimodal feature retrieval library based on the enhanced video features, the enhanced audio features, and the enhanced text features to obtain multimodal retrieval features specifically includes:

[0041] Align and enhance video features Alignment enhances audio features Enhanced text features with alignment The mapped features are then subjected to max pooling to extract the most salient video features. Most prominent audio features and most prominent text features and the most prominent video features Most prominent audio features and most prominent text features Concatenate into a global description vector ;

[0042] Using global description vectors Search the multimodal feature retrieval database to obtain the most similar multimodal samples. .

[0043] Optionally, the emotion recognition method based on multimodal feature retrieval, wherein fusing the weighted enhanced features with the multimodal retrieval features and inputting the fused features into a multimodal large model for processing, and outputting the emotion recognition result of the subject, specifically includes:

[0044] Weighted alignment enhances video features Weighted alignment enhances audio features Weighted alignment enhances text features and multimodal samples After fusion, the data is input into a pre-trained multimodal large model;

[0045] The output results of the multimodal large model According to the results Calculate the classification probability, and output the emotion recognition result of the subject based on the classification probability.

[0046] Optionally, the emotion recognition method based on multimodal feature retrieval further includes:

[0047] During the training phase of the multimodal large model, a total loss function is designed. :

[0048] ;

[0049] in, Loss is categorized by emotion type. Single-modal emotion classification loss, Assign loss to modal weights, and This is a hyperparameter.

[0050] Furthermore, to achieve the above objectives, the present invention also provides an emotion recognition system based on multimodal feature retrieval, wherein the emotion recognition system based on multimodal feature retrieval includes:

[0051] The signal acquisition and feature extraction module is used to acquire video and audio signals of the subject, convert the audio signal into a text signal, and input the video signal, audio signal and text signal into the corresponding feature extraction module to obtain video features, audio features and text features;

[0052] The single-modal feature retrieval enhancement module is used to perform retrieval in a pre-built single-modal feature retrieval library based on the video features, the audio features, and the text features to obtain enhanced video features, enhanced audio features, and enhanced text features;

[0053] A dual-branch processing module is used to align the enhanced video features, enhanced audio features, and enhanced text features to a unified feature space via a mapping module and input them into a dual-branch structure. The modal weight allocator dynamically adjusts the weights of the enhanced video features, enhanced audio features, and enhanced text features to obtain weighted enhanced features. Based on the enhanced video features, enhanced audio features, and enhanced text features, a query is performed in a pre-built multimodal feature retrieval library to obtain multimodal retrieval features.

[0054] The emotion recognition module is used to fuse the weighted enhanced features with the multimodal retrieval features and input them into the multimodal large model for processing, and output the emotion recognition result of the subject.

[0055] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an emotion recognition program based on multimodal feature retrieval stored in the memory and executable on the processor, wherein when the emotion recognition program based on multimodal feature retrieval is executed by the processor, it implements the steps of the emotion recognition method based on multimodal feature retrieval as described above.

[0056] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an emotion recognition program based on multimodal feature retrieval, and the emotion recognition program based on multimodal feature retrieval, when executed by a processor, implements the steps of the emotion recognition method based on multimodal feature retrieval as described above.

[0057] In this invention, video and audio signals of the subject are collected. The audio signal is converted into a text signal. The video, audio, and text signals are then input into corresponding feature extraction modules to obtain video, audio, and text features, respectively. Based on the video, audio, and text features, a pre-built single-modal feature retrieval library is used to obtain enhanced video, audio, and text features. These enhanced features are then aligned to a unified feature space via a mapping module and input into a dual-branch structure. The weights of the enhanced video, audio, and text features are dynamically adjusted by a modal weight allocator to obtain weighted enhanced features. These enhanced features are then queried in a pre-built multimodal feature retrieval library to obtain multimodal retrieval features. Finally, the weighted enhanced features and the multimodal retrieval features are fused and input into a multimodal large-scale model for processing, outputting the subject's emotion recognition result. This invention, by integrating multimodal information such as vision, text, and speech, and combining similar emotion sample retrieval and feature enhancement technologies, can effectively improve the accuracy and robustness of emotion recognition, providing technical support for medical monitoring, nursing management, and human-computer interaction, and realizing accurate perception and intelligent response to individual emotional states. Attached Figure Description

[0058] Figure 1 This is a flowchart of a preferred embodiment of the emotion recognition method based on multimodal feature retrieval of the present invention;

[0059] Figure 2 This is a schematic diagram of the emotion recognition process in a preferred embodiment of the emotion recognition method based on multimodal feature retrieval of the present invention;

[0060] Figure 3 This is a schematic diagram illustrating feature retrieval of an input multimodal signal in a preferred embodiment of the emotion recognition method based on multimodal feature retrieval according to the present invention.

[0061] Figure 4 This is a schematic diagram illustrating the dynamic adjustment of enhancement feature weights via a modal weight allocator in a preferred embodiment of the emotion recognition method based on multimodal feature retrieval of the present invention.

[0062] Figure 5 This is a schematic diagram illustrating multimodal retrieval enhancement in a preferred embodiment of the emotion recognition method based on multimodal feature retrieval according to the present invention;

[0063] Figure 6 This is a structural diagram of a preferred embodiment of the emotion recognition system based on multimodal feature retrieval of the present invention;

[0064] Figure 7This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0066] In multimodal emotion recognition, data from different modalities often exhibit quality differences and inconsistencies. For example, facial images are affected by occlusion and angle changes, speech is affected by noise and accents, and text contains semantic ambiguity or inconsistencies with other modalities. Furthermore, temporal asynchrony, missing data, or differences in emotional intensity may exist between modalities, leading to conflicts during cross-modal feature fusion and reducing recognition accuracy and stability. To address these issues, robust feature representation can be enhanced within a modality by retrieving features; a mapping layer with consistency loss can be designed in cross-modal modeling to ensure emotional consistency of features across different modalities during alignment; furthermore, an adaptive modality weight mechanism can be combined to mitigate information conflicts between modalities, and feature completion can enhance the model's robustness in cases of missing modalities, thereby improving the overall accuracy, balance, and robustness of emotion recognition.

[0067] Compared with traditional unimodal and multimodal emotion recognition methods, this invention can effectively solve the problems in multimodal emotion recognition, such as the inability to effectively identify the emotions of subjects due to differences and inconsistencies in the quality of information within modalities and difficulties in intermodal alignment. It introduces a retrieval mechanism to enhance the expression of features within modalities and intermodal alignment to assist the multimodal model in completing emotion recognition.

[0068] The preferred embodiment of the emotion recognition method based on multimodal feature retrieval described in this invention, such as... Figure 1 and Figure 2 As shown, the emotion recognition method based on multimodal feature retrieval includes the following steps:

[0069] Step S10: Collect video and audio signals from the subject, convert the audio signals into text signals, and input the video signals, audio signals, and text signals into the corresponding feature extraction modules to obtain video features, audio features, and text features.

[0070] Specifically, firstly, features are extracted using existing publicly available datasets and their corresponding encoders. Based on these extracted features, a single-modal feature retrieval library and a multi-modal feature retrieval library are constructed. Video signals from the subjects are then acquired using a camera and microphone. (Video signal used to extract facial information) and audio signal The audio signal The speech recognition module converts the speech signal into a text signal. Thus, video signals, audio signals, and text signals were obtained. Then the video signal Audio signals With text signals Input the video encoder, audio encoder, and text encoder respectively to obtain video features. Audio features Text features (Right now ).

[0071] Step S20: Search the pre-built single-modal feature retrieval library according to the video features, audio features and text features to obtain enhanced video features, enhanced audio features and enhanced text features.

[0072] Specifically, various features are retrieved from a pre-built single-modal feature retrieval library (i.e., through...). Figure 2 The single-modal retrieval enhancement module in the system performs retrieval to achieve feature enhancement, such as... Figure 3 As shown, in order to enable the multimodal large model to effectively utilize prior retrieval knowledge to enhance classification robustness and improve performance, feature retrieval needs to be performed on the input multimodal signal. This process first uses a pre-trained encoder to extract the input video signal, audio signal, and text signal respectively. The corresponding three modal features were obtained. ,Right now Subsequently, cosine similarity was calculated and searched in dedicated retrieval databases for each modality. K - Nearest neighbor search (given a point, quickly find the nearest neighbor in the dataset) K Each input sample has "neighbors" (number of "neighbors"). Accurately retrieve the most similar previous K candidate samples , and Marked as intra-modal retrieval features To address the efficiency challenges of large-scale data, the FAISS library (FAISS is an open-source library developed by the Facebook AI Research team, specifically designed for efficient large-scale vector similarity search, especially for dense vector similarity search problems) is used for high-speed vector indexing and approximate nearest neighbor search to improve retrieval speed and system scalability.

[0073] For the original feature and the retrieved candidate samples within the corresponding modality The process of retrieving intra-modal features using bidirectional attention is as follows:

[0074] First, perform positive cross-attention, and transfer the original features... and intramodal retrieval features Perform linear mapping to generate positive cross-attention queries respectively. Positive cross attention key and positive cross attention value Furthermore, the positive cross-attention weights of the original features and the intra-modal retrieval features are calculated. :

[0075] ;

[0076] in, This is the scaling factor. For the Softmax function, This indicates transpose.

[0077] Furthermore, the product of the positive cross-attention value and the positive cross-attention weight is calculated to obtain the features enhanced by positive cross-attention retrieval. :

[0078] .

[0079] Similarly, intra-modal retrieval features and original features Perform linear mapping to generate reverse cross-attention queries respectively. Reverse cross attention key and reverse cross attention value Calculate the inverse cross-attention weights between the retrieved features and the original features within the modality. :

[0080] ;

[0081] Then, the product of the back cross-attention value and the back cross-attention weight is calculated to obtain the feature enhanced by back cross-attention retrieval. :

[0082] .

[0083] Further enhance the features retrieved by positive cross-attention. Enhanced features retrieved via reverse cross-attention The features are added together, and the residuals are then concatenated with the original features to create a single-modal augmentation feature. Among them, single-mode enhancement features Includes enhanced video features Enhance audio features Enhanced text features (Right now ).

[0084] To ensure the enhanced single-modal enhancement features The effectiveness of using classification loss to enhance single-modality features. Apply constraints:

[0085] ;

[0086] in, For unimodal emotion classification loss, For visual modality emotion classification loss, For audio modality emotion classification loss, The loss is used for text modality sentiment classification.

[0087] Step S30: Align the enhanced video features, enhanced audio features, and enhanced text features to a unified feature space via a mapping module, and input them into a dual-branch structure. Dynamically adjust the weights of the enhanced video features, enhanced audio features, and enhanced text features through a modal weight allocator to obtain weighted enhanced features. Query the enhanced video features, enhanced audio features, and enhanced text features in a pre-built multimodal feature retrieval library to obtain multimodal retrieval features.

[0088] Specifically, in multimodal emotion recognition (MER) systems (an AI system capable of simultaneously analyzing and fusing information from multiple sources such as language, facial expressions, voice, and physiological signals to more accurately identify and understand human emotional states), different modalities often exhibit significant heterogeneity and differences in information expression. For example, in some samples, the text modality may contain richer emotional semantics (e.g., "very happy"), while in other samples, facial expressions or tone of voice may be more discriminative (e.g., a noticeable smile or an excited tone). If a static or fixed-weight fusion strategy is used, the model is prone to over-reliance on a dominant modality, neglecting supplementary information from other modalities, leading to a decline in the model's generalization ability. This modality imbalance problem is particularly prominent: on the one hand, the dominant modality suppresses the learning of other modalities; on the other hand, noisy modalities (e.g., occluded faces or low-quality audio) may introduce interference, reducing overall recognition performance. Therefore, introducing a dynamic modality balancing mechanism is crucial.

[0089] like Figure 4 As shown, the first step is to enhance video features. Enhance audio features Enhanced text features (Right now Feature mapping is performed, and the features are projected onto a unified feature space through a multi-layer perceptron (MLP) to obtain aligned and enhanced video features. Alignment enhances audio features Enhanced text features with alignment (Right now This achieves feature alignment across video, audio, and text modalities, ensuring semantic consistency across different modalities and laying the foundation for subsequent fusion. To further fuse multimodal features, the alignment enhances video features. Alignment enhances audio features Enhanced text features with alignment Input to the modal weight assigner (i.e. Figure 2 (The modal importance allocation module in the middle).

[0090] Specifically, the modality weight allocator dynamically calculates the modality importance weight distribution for each sample through a routing network consisting of a linear layer, batch normalization, and activation functions. , Represents video modal weights. Represents audio modal weights. This represents the text modality weights. After calculating the modality weight coefficients, the alignment enhancement video features are then applied based on the modality importance weight distribution for each sample. Alignment enhances audio features Enhanced text features with alignment After correction, the weighted enhanced features (i.e.) are obtained. ):

[0091] ;

[0092] ;

[0093] ;

[0094] The weighted enhancement features include weighted alignment enhancement video features. Weighted alignment enhances audio features Enhanced text features with weighted alignment .

[0095] To avoid a single modality dominating the multimodal feature fusion process, a modality weight allocation loss is constructed. The definition is as follows:

[0096] ;

[0097] in, Representing modes The assigned weights, including , and When the entropy loss is minimized, the weight distribution tends to be uniform; however, if the weight of a certain mode is close to 1, the entropy loss will increase significantly, thereby effectively suppressing the mode monopoly phenomenon, prompting the model to make full use of the complementary information of each mode, and enhancing the discriminative ability of multimodal representation.

[0098] like Figure 5 As shown, after completing the feature mapping of video, audio, and text modal information, a multimodal retrieval enhancement mechanism is introduced to incorporate global information from an external knowledge base, thereby improving the robustness of emotion recognition. This is the other branch in the dual-branch structure. Figure 2 The multimodal retrieval enhancement module in the text. Specifically, it first aligns and enhances video features. Alignment enhances audio features Enhanced text features with alignment (Right now The mapped features are then subjected to max pooling to extract the most salient video features. Most prominent audio features and most prominent text features and the most prominent video features Most prominent audio features and most prominent text features Concatenate into a global description vector Then, the global description vector is used. Search the multimodal feature retrieval database to obtain the most similar multimodal samples. .

[0099] Step S40: After fusing the weighted enhanced features with the multimodal retrieval features, input the fusion into the multimodal large model for processing, and output the emotion recognition result of the subject.

[0100] Specifically, weighted alignment enhances video features. Weighted alignment enhances audio features Weighted alignment enhances text features and multimodal samples After fusion, the data is input into a pre-trained multimodal large model; here, the following method is used. Figure 5 LoRA (Low-Rank Adaptation) is used for fine-tuning. By freezing the main weights of a large multimodal model, only a pair of low-rank matrices are introduced and trained to approximate the effect of full parameter fine-tuning.

[0101] After calculation, the multimodal large model outputs the results. According to the results Calculate the classification probability, and output the emotion recognition result of the subject based on the classification probability.

[0102] Furthermore, the output results of the multimodal large model are... By connecting to a linear layer, the dimensions of the enhanced features are mapped to seven categories (corresponding to seven emotion categories, such as...). Figure 3 and Figure 5 As shown, seven emotion categories are identified (neutral, anger, sadness, happiness, disgust, fear, and surprise). The Softmax function is then used to calculate the value for each dimension, representing the probability that the subject belongs to each emotion at the current moment. To enhance the robustness and effectiveness of each modality's feature representation and to prevent a single modality from dominating the fusion process, a total loss function is designed during the training phase of the multimodal large model. :

[0103] ;

[0104] in, Loss is categorized by emotion type. Single-modal emotion classification loss, Assign loss to modal weights, and This is a hyperparameter, and its value here is 1.

[0105] This invention first constructs a unimodal and multimodal feature retrieval library by extracting features from existing public datasets and their corresponding encoders. It processes video, audio, and text converted from audio to form a trimodal input. After feature extraction from each modality, the data is retrieved through the unimodal feature retrieval library to achieve preliminary feature enhancement. The enhanced features are then aligned to a unified feature space via a mapping module and subsequently input into a dual-branch structure: one branch dynamically adjusts the weights of the enhanced features across modalities using a modal weight allocator, while the other branch uses the enhanced features to query the multimodal feature retrieval library to obtain multimodal retrieval features. Finally, the weighted enhanced features are fused with the multimodal retrieval features and input into a large multimodal model for deep interaction and inference, thereby achieving accurate identification of the subject's emotions.

[0106] Furthermore, such as Figure 6 As shown, based on the above-mentioned emotion recognition method based on multimodal feature retrieval, the present invention also provides an emotion recognition system based on multimodal feature retrieval, wherein the emotion recognition system based on multimodal feature retrieval includes:

[0107] The signal acquisition and feature extraction module 51 is used to acquire video and audio signals of the subject, convert the audio signal into a text signal, and input the video signal, the audio signal and the text signal into the corresponding feature extraction module to obtain video features, audio features and text features;

[0108] The single-modal feature retrieval enhancement module 52 is used to perform retrieval in a pre-built single-modal feature retrieval library based on the video features, the audio features, and the text features to obtain enhanced video features, enhanced audio features, and enhanced text features;

[0109] The dual-branch processing module 53 is used to align the enhanced video features, enhanced audio features, and enhanced text features to a unified feature space via a mapping module and input them into a dual-branch structure. The modality weight allocator dynamically adjusts the weights of the enhanced video features, enhanced audio features, and enhanced text features to obtain weighted enhanced features. Based on the enhanced video features, enhanced audio features, and enhanced text features, the module queries a pre-built multimodal feature retrieval library to obtain multimodal retrieval features.

[0110] The emotion recognition module 54 is used to fuse the weighted enhanced features with the multimodal retrieval features and input them into the multimodal large model for processing, and output the emotion recognition result of the subject.

[0111] Furthermore, such as Figure 6 As shown, based on the above-mentioned emotion recognition method and system based on multimodal feature retrieval, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0112] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores an emotion recognition program 40 based on multimodal feature retrieval, which can be executed by the processor 10 to implement the emotion recognition method based on multimodal feature retrieval in this application.

[0113] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the emotion recognition method based on multimodal feature retrieval.

[0114] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminal's processor 10, memory 20, and display 30 communicate with each other via a system bus.

[0115] In one embodiment, when the processor 10 executes the emotion recognition program 40 based on multimodal feature retrieval in the memory 20, it implements the steps of the emotion recognition method based on multimodal feature retrieval as described above.

[0116] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an emotion recognition program based on multimodal feature retrieval, and the emotion recognition program based on multimodal feature retrieval, when executed by a processor, implements the steps of the emotion recognition method based on multimodal feature retrieval as described above.

[0117] In summary, this invention provides an emotion recognition method, system, terminal, and computer-readable storage medium based on multimodal feature retrieval. The method includes: acquiring video and audio signals from a subject; converting the audio signals into text signals; inputting the video, audio, and text signals into corresponding feature extraction modules to obtain video features, audio features, and text features; searching a pre-constructed single-modal feature retrieval library based on the video, audio, and text features to obtain enhanced video features, enhanced audio features, and enhanced text features; aligning the enhanced video, audio, and text features to a unified feature space via a mapping module and inputting them into a dual-branch structure; dynamically adjusting the weights of the enhanced video, audio, and text features using a modal weight allocator to obtain weighted enhanced features; querying a pre-constructed multimodal feature retrieval library based on the enhanced video, audio, and text features to obtain multimodal retrieval features; fusing the weighted enhanced features and the multimodal retrieval features and inputting them into a multimodal large-scale model for processing; and outputting the emotion recognition result of the subject. This invention, by integrating multimodal information such as vision, text, and speech, and combining similar emotion sample retrieval and feature enhancement technologies, can effectively improve the accuracy and robustness of emotion recognition, providing technical support for medical monitoring, nursing management, and human-computer interaction, and realizing accurate perception and intelligent response to individual emotional states.

[0118] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0119] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0120] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for emotion recognition based on multi-modal feature retrieval, characterized in that, The emotion recognition method based on multimodal feature retrieval includes: Video and audio signals of the subjects are collected, the audio signals are converted into text signals, and the video signals, audio signals and text signals are respectively input into the corresponding feature extraction modules to obtain video features, audio features and text features; Based on the video features, audio features, and text features, a search is performed in a pre-built single-modal feature retrieval library to obtain enhanced video features, enhanced audio features, and enhanced text features; The step of retrieving enhanced video features, enhanced audio features, and enhanced text features from a pre-built single-modal feature retrieval library based on the video features, the audio features, and the text features to obtain enhanced video features, enhanced audio features, and enhanced text features specifically includes: Cosine similarity is calculated and performed in dedicated search databases for each modality. K -Nearest neighbor search retrieves the most similar neighbors for each input sample. K 1 candidate sample, labeled as intra-modal retrieval feature; Linearly map the original features and intramodal retrieval features to generate positive cross-attention queries, positive cross-attention keys, and positive cross-attention values, respectively, and calculate the positive cross-attention weights between the original features and intramodal retrieval features; Calculate the product of the positive cross-attention value and the positive cross-attention weight to obtain the features enhanced by positive cross-attention retrieval; Linearly map the intramodal retrieval features and the original features to generate reverse cross-attention queries, reverse cross-attention keys, and reverse cross-attention values, and calculate the reverse cross-attention weights between the intramodal retrieval features and the original features. Calculate the product of the back cross attention value and the back cross attention weight to obtain the features enhanced by back cross attention retrieval; The features enhanced by forward cross-attention retrieval and the features enhanced by reverse cross-attention retrieval are added together and then residually connected with the original features to form a single-modal enhanced feature. The single-modal enhanced features include enhanced video features, enhanced audio features, and enhanced text features. The enhanced video features, enhanced audio features, and enhanced text features are aligned to a unified feature space via a mapping module and input into a dual-branch structure. The weights of the enhanced video features, enhanced audio features, and enhanced text features are dynamically adjusted by a modal weight allocator to obtain weighted enhanced features. Based on the enhanced video features, enhanced audio features, and enhanced text features, a query is performed in a pre-built multimodal feature retrieval library to obtain multimodal retrieval features. The weighted enhanced features are fused with the multimodal retrieval features and then input into a multimodal large model for processing, outputting the emotion recognition results of the subject.

2. The emotion recognition method based on multimodal feature retrieval according to claim 1, characterized in that, The step of retrieving enhanced video features, enhanced audio features, and enhanced text features from a pre-built single-modal feature retrieval library based on the video features, the audio features, and the text features, and prior to this step, also includes: Features are extracted in advance using existing public datasets and corresponding encoders, and single-modal feature retrieval libraries and multimodal feature retrieval libraries are constructed based on the extracted features.

3. The emotion recognition method based on multimodal feature retrieval according to claim 1, characterized in that, The step of inputting the video signal, the audio signal, and the text signal into the corresponding feature extraction modules to obtain video features, audio features, and text features specifically involves: video signal Audio signals With text signals Input the video encoder, audio encoder, and text encoder respectively to obtain video features. Audio features Text features .

4. The emotion recognition method based on multimodal feature retrieval according to claim 3, characterized in that, The step of retrieving enhanced video features, enhanced audio features, and enhanced text features from a pre-built single-modal feature retrieval library based on the video features, the audio features, and the text features to obtain enhanced video features, enhanced audio features, and enhanced text features specifically includes: Cosine similarity is calculated and performed in dedicated search databases for each modality. K -Nearest neighbor search, for each input sample Retrieve the most similar ones K candidate samples , and Marked as intra-modal retrieval features ; Original features and intramodal retrieval features Perform linear mapping to generate positive cross-attention queries respectively. Positive cross attention key and positive cross attention value Calculate the positive cross-attention weights between the original features and the in-modal retrieval features. : ； wherein, is a scale factor, is a Softmax function, denotes a transpose; The product of the forward cross-attention value and the forward cross-attention weight is calculated to obtain a feature enhanced by forward cross-attention retrieval : ； Intra-modal retrieval features and original features are linearly mapped to generate reverse cross-attention query , reverse cross-attention key and reverse cross-attention value respectively, and reverse cross-attention weights between intra-modal retrieval features and original features are calculated as follows : ； The product of the reverse cross-attention value and the reverse cross-attention weight is calculated to obtain a feature enhanced by reverse cross-attention retrieval : ； Enhanced features retrieved using positive cross-attention. Features enhanced by reverse cross-attention retrieval The features are added together, and the residuals are then concatenated with the original features to create a single-modal augmentation feature. Among them, single-mode enhancement features Includes enhanced video features Enhance audio features Enhanced text features ; Enhance features for unimodal use of classification loss. Constraints are imposed to ensure single-modal enhancement features. Validity: ； in, For unimodal emotion classification loss, For visual modality emotion classification loss, For audio modality emotion classification loss, The loss is used for text modality sentiment classification. 5.The emotion recognition method based on multi-modal feature retrieval according to claim 4, characterized in that, The process of aligning the enhanced video features, enhanced audio features, and enhanced text features to a unified feature space via a mapping module, inputting them into a dual-branch structure, and dynamically adjusting the weights of the enhanced video features, enhanced audio features, and enhanced text features through a modal weight allocator to obtain weighted enhanced features specifically includes: Enhanced video features Enhance audio features Enhanced text features Feature mapping is performed, and the data is projected onto a unified feature space through a multilayer perceptron to obtain aligned and enhanced video features. Alignment enhances audio features Enhanced text features with alignment This will align and enhance video features. Alignment enhances audio features Enhanced text features with alignment Input to the modal weight assigner; The modality weight allocator dynamically calculates the modality importance weight distribution for each sample through a routing network consisting of a linear layer, batch normalization, and activation functions. , Represents video modal weights. Represents audio modal weights. Representing text modality weights, this method aligns and enhances video features based on the modality importance weight distribution for each sample. Alignment enhances audio features Enhanced text features with alignment After correction, the weighted enhanced features are obtained: ； ； ； The weighted enhanced features include weighted aligned enhanced video features , weighted aligned enhanced audio features , and weighted aligned enhanced text features ​ constructing a modal weight distribution loss : ； wherein, represents a modality assigned weight, including , and ; The step of querying a pre-built multimodal feature retrieval library based on the enhanced video features, the enhanced audio features, and the enhanced text features to obtain multimodal retrieval features specifically includes: Align and enhance video features Alignment enhances audio features Enhanced text features with alignment The mapped features are then subjected to max pooling to extract the most salient video features. Most prominent audio features and most prominent text features and the most prominent video features Most prominent audio features and most prominent text features Concatenate into a global description vector ; Using global description vectors Search the multimodal feature retrieval database to obtain the most similar multimodal samples. .

6. The emotion recognition method based on multimodal feature retrieval according to claim 5, characterized in that, The process of fusing the weighted enhanced features with the multimodal retrieval features and then inputting the result into a multimodal large model for processing, and outputting the emotion recognition result of the subject, specifically includes: Weighted alignment enhances video features Weighted alignment enhances audio features Weighted alignment enhances text features and multimodal samples After fusion, the data is input into a pre-trained multimodal large model; The multi-modal large model outputs a result , according to the result computes a classification probability, and outputs an emotion recognition result of the subject according to the classification probability.

7. The emotion recognition method based on multi-modal feature retrieval according to any one of claims 1-6, characterized in that, The emotion recognition method based on multimodal feature retrieval also includes: In the training phase of the multi-modal large model, a total loss function is designed : ； in, Loss is categorized by emotion type. Single-modal emotion classification loss, Assign loss to modal weights, and This is a hyperparameter.

8. An emotion recognition system based on multi-modal feature retrieval, characterized in that, The emotion recognition system based on multimodal feature retrieval is used to implement the emotion recognition method based on multimodal feature retrieval according to any one of claims 1-7, wherein the emotion recognition system based on multimodal feature retrieval includes: The signal acquisition and feature extraction module is used to acquire video and audio signals of the subject, convert the audio signal into a text signal, and input the video signal, audio signal and text signal into the corresponding feature extraction module to obtain video features, audio features and text features; The single-modal feature retrieval enhancement module is used to perform retrieval in a pre-built single-modal feature retrieval library based on the video features, the audio features, and the text features to obtain enhanced video features, enhanced audio features, and enhanced text features; A dual-branch processing module is used to align the enhanced video features, enhanced audio features, and enhanced text features to a unified feature space via a mapping module and input them into a dual-branch structure. The modal weight allocator dynamically adjusts the weights of the enhanced video features, enhanced audio features, and enhanced text features to obtain weighted enhanced features. Based on the enhanced video features, enhanced audio features, and enhanced text features, a query is performed in a pre-built multimodal feature retrieval library to obtain multimodal retrieval features. The emotion recognition module is used to fuse the weighted enhanced features with the multimodal retrieval features and input them into the multimodal large model for processing, and output the emotion recognition result of the subject.

9. A terminal, characterized by comprising: The terminal includes: a memory, a processor, and an emotion recognition program based on multimodal feature retrieval stored in the memory and executable on the processor. When the emotion recognition program based on multimodal feature retrieval is executed by the processor, it implements the steps of the emotion recognition method based on multimodal feature retrieval as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores an emotion recognition program based on multi-modal feature retrieval, and the emotion recognition program based on multi-modal feature retrieval, when executed by the processor, implements the steps of the emotion recognition method based on multi-modal feature retrieval as claimed in any one of claims 1-7.

Images