Emotion recognition method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an emotion recognition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring multimedia data of a target object, and inputting the multimedia data into an emotion recognition model for emotion recognition to obtain an emotion recognition result. Wherein the emotion recognition model comprises a feature extraction unit, a feature fusion unit and an output unit, the feature extraction unit is used for extracting feature data of at least two modalities from multimedia data, and the feature fusion unit is used for performing pairwise fusion on the feature data of the at least two modalities by adopting at least one cross attention mechanism, the output unit is used for outputting an emotion recognition result based on the at least one fusion feature, and each cross attention mechanism is matched with a preset modal combination. By adopting the method, the accuracy of emotion recognition can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] Emotion recognition is a key branch of artificial intelligence, deeply integrating technologies from natural language processing, computer vision, machine learning, and other fields. Currently, multimodal fusion has become a core research hotspot in the field of emotion recognition. By integrating information from different dimensions such as vision, speech, and text, it can effectively compensate for the limitations of single-modal recognition and improve the comprehensiveness and accuracy of emotion capture.

[0003] In related technologies, traditional multimodal fusion emotion recognition technology first extracts visual features using CNN (Convolutional Neural Network), extracts speech features using LSTM (Long Short-Term Memory), and extracts text features using BERT (Bidirectional Encoder Representations from Transformers). After normalization, vectors are concatenated or weighted summed with fixed weights. After fusion, emotion recognition is achieved by direct classification using SVM (Support Vector Machine) or a fully connected layer with batch normalization and activation functions.

[0004] However, the relevant technologies may have the following problems: Traditional multimodal fusion technology performs feature fusion by simply splicing or by weighted summation with fixed weights. This method cannot explore the collaborative relationship between modalities, which leads to poor accuracy in emotion recognition. Summary of the Invention

[0005] Therefore, it is necessary to provide an emotion recognition method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of emotion recognition in response to the above-mentioned technical problems.

[0006] Firstly, this application provides an emotion recognition method, including:

[0007] Obtain multimedia data of the target object;

[0008] The multimedia data is input into the emotion recognition model for emotion recognition, and the emotion recognition result of the target object is obtained.

[0009] The emotion recognition model includes a feature extraction unit, a feature fusion unit, and an output unit. The feature extraction unit is used to extract feature data of at least two modalities from the multimedia data. The feature fusion unit is constructed based on at least one cross-attention mechanism. The feature fusion unit is used to fuse the feature data of the at least two modalities pairwise using the at least one cross-attention mechanism to obtain at least one fused feature. The output unit is used to output the corresponding emotion recognition result based on the at least one fused feature. Each cross-attention mechanism is matched with a preset modal combination.

[0010] In one embodiment, the feature extraction unit includes at least two of a first feature extraction subunit, a second feature extraction subunit, and a third feature extraction subunit. The process by which the feature extraction unit extracts feature data of at least two modalities from the multimedia data includes:

[0011] The first feature extraction subunit extracts visual features from the video frames of the multimedia data;

[0012] The second feature extraction subunit extracts audio features from the audio data of the multimedia data;

[0013] The third feature extraction subunit converts the audio data into text data and extracts text features from the text data.

[0014] In one embodiment, the feature data of the at least two modalities are paired up to obtain n feature data groups. The feature fusion unit includes n cross-attention mechanisms, which are used to fuse the feature data in the n feature data groups, where n is a positive integer.

[0015] In one embodiment, the feature data set includes first modality feature data and second modality feature data. The cross-attention mechanism fuses the feature data in the feature data set, including:

[0016] Based on the emotion query intent represented by the first modality feature data, the feature association information between the second modality feature data and the emotion query intent is determined, and the second modality feature data is subjected to feature filtering and fusion for the emotion query intent based on the feature association information to obtain the corresponding fused features.

[0017] In one embodiment, the step of determining feature association information between the second modality feature data and the emotion query intent represented by the first modality feature data, and performing feature filtering and fusion on the second modality feature data based on the feature association information for the emotion query intent to obtain corresponding fused features, includes:

[0018] A query feature vector is obtained by linearly projecting the first modality feature data onto the query weight matrix. The query feature vector is used to characterize the sentiment query intent of the first modality feature data.

[0019] A key feature vector is obtained by linearly projecting the second modality feature data through a key weight matrix. The key feature vector includes feature information for matching the sentiment query intent.

[0020] The second modality feature data is linearly projected onto the value weight matrix to obtain the value feature vector, which is the emotion feature used to carry the second modality feature data;

[0021] Based on the query feature vector and the key feature vector, a feature association weight matrix is ​​generated between the first modality feature data and the second modality feature data, and a corresponding fusion feature is generated based on the feature association weight matrix and the value feature vector.

[0022] In one embodiment, the training process of the emotion recognition model includes:

[0023] A training set is constructed, which includes at least one training subset. Each training subset corresponds to one cross-attention mechanism. For any cross-attention mechanism, the cross-attention mechanism is used to perform feature fusion on a matched preset modality combination. The preset modality combination includes a first modality and a second modality. The training subset corresponding to the cross-attention mechanism includes positive sample pairs and negative sample pairs. The ground truth sentiments of the two sample data in the positive sample pair are the same, and the ground truth sentiments of the two sample data in the negative sample pair are different. Furthermore, the two sample data represent the same sentiment in the feature data of the first modality, but different sentiments in the feature data of the second modality.

[0024] The initial emotion recognition model is used to perform emotion recognition on each of the sample data to obtain the emotion recognition result corresponding to each of the sample data, as well as the fusion features output by each of the cross-attention mechanisms for each of the sample data;

[0025] The initial emotion recognition model is trained based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, and the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs.

[0026] Model training is completed when the preset training stopping conditions are met.

[0027] In one embodiment, training the initial emotion recognition model based on the emotion recognition result, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, and the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, includes:

[0028] Based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, and the joint loss function, the total recognition loss of the initial emotion recognition model is determined. The joint loss function is a loss function constructed with the training objective of simultaneously optimizing the emotion recognition accuracy and the feature fusion discrimination of the cross-attention mechanism.

[0029] The initial emotion recognition model is tuned based on the total recognition loss.

[0030] In one embodiment, the total recognition loss of the initial emotion recognition model is determined based on the emotion recognition result, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, and the joint loss function, including:

[0031] Based on the emotion recognition results corresponding to each of the sample data, the emotion recognition loss of the initial emotion recognition model is determined.

[0032] For any of the aforementioned cross-attention mechanisms, the feature fusion loss of the cross-attention mechanism is determined based on the fusion features of each sample data in the positive sample pairs in the training subset corresponding to the cross-attention mechanism and the fusion features of each sample data in the negative sample pairs.

[0033] The total recognition loss of the initial emotion recognition model is determined based on the emotion recognition loss and the feature fusion loss of each of the cross-attention mechanisms.

[0034] Secondly, this application also provides an emotion recognition device, comprising:

[0035] The acquisition module is used to acquire multimedia data of the target object;

[0036] The recognition module is used to input the multimedia data into the emotion recognition model for emotion recognition and obtain the emotion recognition result of the target object;

[0037] The emotion recognition model includes a feature extraction unit, a feature fusion unit, and an output unit. The feature extraction unit is used to extract feature data of at least two modalities from the multimedia data. The feature fusion unit is constructed based on at least one cross-attention mechanism. The feature fusion unit is used to fuse the feature data of the at least two modalities pairwise using the at least one cross-attention mechanism to obtain at least one fused feature. The output unit is used to output the corresponding emotion recognition result based on the at least one fused feature. Each cross-attention mechanism is matched with a preset modal combination.

[0038] In one embodiment, the feature extraction unit includes at least two of a first feature extraction subunit, a second feature extraction subunit, and a third feature extraction subunit. The process by which the feature extraction unit extracts feature data of at least two modalities from the multimedia data includes:

[0039] The first feature extraction subunit extracts visual features from the video frames of the multimedia data;

[0040] The second feature extraction subunit extracts audio features from the audio data of the multimedia data;

[0041] The third feature extraction subunit converts the audio data into text data and extracts text features from the text data.

[0042] In one embodiment, the feature data of the at least two modalities are paired up to obtain n feature data groups. The feature fusion unit includes n cross-attention mechanisms, which are used to fuse the feature data in the n feature data groups, where n is a positive integer.

[0043] In one embodiment, the feature data set includes first modality feature data and second modality feature data. The cross-attention mechanism fuses the feature data in the feature data set, including:

[0044] Based on the emotion query intent represented by the first modality feature data, the feature association information between the second modality feature data and the emotion query intent is determined, and the second modality feature data is subjected to feature filtering and fusion for the emotion query intent based on the feature association information to obtain the corresponding fused features.

[0045] In one embodiment, the step of determining feature association information between the second modality feature data and the emotion query intent represented by the first modality feature data, and performing feature filtering and fusion on the second modality feature data based on the feature association information for the emotion query intent to obtain corresponding fused features, includes:

[0046] A query feature vector is obtained by linearly projecting the first modality feature data onto the query weight matrix. The query feature vector is used to characterize the sentiment query intent of the first modality feature data.

[0047] A key feature vector is obtained by linearly projecting the second modality feature data through a key weight matrix. The key feature vector includes feature information for matching the sentiment query intent.

[0048] The second modality feature data is linearly projected onto the value weight matrix to obtain the value feature vector, which is the emotion feature used to carry the second modality feature data;

[0049] Based on the query feature vector and the key feature vector, a feature association weight matrix is ​​generated between the first modality feature data and the second modality feature data, and a corresponding fusion feature is generated based on the feature association weight matrix and the value feature vector.

[0050] In one embodiment, the training process of the emotion recognition model includes:

[0051] A training set is constructed, which includes at least one training subset. Each training subset corresponds to one cross-attention mechanism. For any cross-attention mechanism, the cross-attention mechanism is used to perform feature fusion on a matched preset modality combination. The preset modality combination includes a first modality and a second modality. The training subset corresponding to the cross-attention mechanism includes positive sample pairs and negative sample pairs. The ground truth sentiments of the two sample data in the positive sample pair are the same, and the ground truth sentiments of the two sample data in the negative sample pair are different. Furthermore, the two sample data represent the same sentiment in the feature data of the first modality, but different sentiments in the feature data of the second modality.

[0052] The initial emotion recognition model is used to perform emotion recognition on each of the sample data to obtain the emotion recognition result corresponding to each of the sample data, as well as the fusion features output by each of the cross-attention mechanisms for each of the sample data;

[0053] The initial emotion recognition model is trained based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, and the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs.

[0054] Model training is completed when the preset training stopping conditions are met.

[0055] In one embodiment, training the initial emotion recognition model based on the emotion recognition result, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, and the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, includes:

[0056] Based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, and the joint loss function, the total recognition loss of the initial emotion recognition model is determined. The joint loss function is a loss function constructed with the training objective of simultaneously optimizing the emotion recognition accuracy and the feature fusion discrimination of the cross-attention mechanism.

[0057] The initial emotion recognition model is tuned based on the total recognition loss.

[0058] In one embodiment, the total recognition loss of the initial emotion recognition model is determined based on the emotion recognition result, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, and the joint loss function, including:

[0059] Based on the emotion recognition results corresponding to each of the sample data, the emotion recognition loss of the initial emotion recognition model is determined.

[0060] For any of the aforementioned cross-attention mechanisms, the feature fusion loss of the cross-attention mechanism is determined based on the fusion features of each sample data in the positive sample pairs in the training subset corresponding to the cross-attention mechanism and the fusion features of each sample data in the negative sample pairs.

[0061] The total recognition loss of the initial emotion recognition model is determined based on the emotion recognition loss and the feature fusion loss of each of the cross-attention mechanisms.

[0062] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above-mentioned embodiments.

[0063] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0064] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0065] The aforementioned emotion recognition method, apparatus, computer device, computer-readable storage medium, and computer program product acquire multimedia data of a target object and input the multimedia data into an emotion recognition model for emotion recognition, thereby obtaining the emotion recognition result of the target object. The emotion recognition model includes a feature extraction unit, a feature fusion unit, and an output unit. The feature extraction unit extracts feature data from at least two modalities from the multimedia data. The feature fusion unit is constructed based on at least one cross-attention mechanism and uses at least one cross-attention mechanism to fuse the feature data of at least two modalities pairwise to obtain at least one fused feature. The output unit outputs the corresponding emotion recognition result based on the at least one fused feature. Each cross-attention mechanism is matched with a preset modality combination. Using the emotion recognition method, apparatus, computer device, computer-readable storage medium, and computer program product provided in this application embodiment, in an emotion recognition scenario, the emotion recognition model extracts multimodal features and achieves effective fusion between multimodal features through cross-attention mechanisms. By fusing features, it discovers the synergistic relationship between multimodal features, thereby reducing the false positive rate of emotion recognition and effectively improving the accuracy of emotion recognition. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1This is a flowchart illustrating an emotion recognition method in one embodiment;

[0068] Figure 2 This is a flowchart illustrating the feature data extraction process in one embodiment;

[0069] Figure 3 This is a flowchart illustrating the feature data fusion process in one embodiment;

[0070] Figure 4 This is a flowchart illustrating the model training process in one embodiment;

[0071] Figure 5 This is a flowchart illustrating step 406 in one embodiment;

[0072] Figure 6 This is a flowchart illustrating the model training process in another embodiment;

[0073] Figure 7 This is a schematic diagram of an emotion recognition method in one embodiment;

[0074] Figure 8 This is a structural block diagram of an emotion recognition device in one embodiment;

[0075] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0077] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0078] The user data, data acquisition, and / or use involved in the embodiments of this application strictly comply with the laws, regulations, and industry standards of relevant countries and regions. The collection and acquisition of data involved in the embodiments of this application are all done in advance by actively prompting or prominently displaying information to inform users and obtaining authorization, or by obtaining full authorization from all parties. The processing, manipulation, forwarding, and use of data involved in the embodiments of this application are all carried out on the premise that the user or relevant party is fully informed and authorized. In implementing the various embodiments of this application, the types of data or information, scope of use, and usage scenarios that may be involved are informed to users or relevant parties and authorization is obtained through appropriate means. The specific methods of notification and authorization may vary according to actual circumstances, and this application is not limited in this regard. The processing of personal information involved in the embodiments of this application is carried out under the premise of having a legal basis (such as obtaining the consent of the personal information subject or being necessary for the performance of a contract), and is only processed within the prescribed or agreed scope. Sensitive personal information such as biometric information, medical and health information, financial account information, and precise location information involved in the embodiments of this application are all processed under the premise of having a specific purpose and sufficient necessity, and with the separate authorization and consent of the user or relevant party.

[0079] In some embodiments of this application, if the user or related party refuses to process personal information other than the information necessary for the basic functions, it will not affect the use of the basic functions of the embodiments of this application.

[0080] In one embodiment, such as Figure 1 As shown, an emotion recognition method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes steps 102 to 104, wherein:

[0081] Step 102: Obtain the multimedia data of the target object.

[0082] In this embodiment, multimedia data is a multimodal information carrier containing the emotional expression of the target object, and may include video data and audio data, etc. Specifically, video data includes multimedia data containing both visual images and sound, and audio data is multimedia data containing sound; audio data can be further converted into text information.

[0083] For example, the target object can be a natural person, and multimedia data can be collected in real time through terminal devices such as cameras and microphones, or read from storage media. Video data records the target object's facial expressions, such as the curvature of the corners of the mouth, changes in eye contact, and visual behavioral cues like body movements. Audio data contains the target object's speech and intonation, such as speech rate, pitch, and timbre variations. Furthermore, speech-to-text technologies like Whisper can extract textual information from the audio data. This textual information contains semantic connotations of the language content, such as emotional words like "happy" or "sad."

[0084] Step 104: Input the multimedia data into the emotion recognition model for emotion recognition to obtain the emotion recognition result of the target object. The emotion recognition model includes a feature extraction unit, a feature fusion unit, and an output unit. The feature extraction unit is used to extract feature data of at least two modalities from the multimedia data. The feature fusion unit is constructed based on at least one cross-attention mechanism. The feature fusion unit is used to fuse the feature data of at least two modalities pairwise using at least one cross-attention mechanism to obtain at least one fused feature. The output unit is used to output the corresponding emotion recognition result based on at least one fused feature. Each cross-attention mechanism is matched with a preset modal combination.

[0085] In this embodiment, the emotion recognition model is a model that achieves accurate emotion recognition based on multimodal information. It uses a three-level architecture, consisting of a feature extraction unit for feature extraction, a feature fusion unit for cross-modal feature fusion, and an output unit for classification output, to convert multimedia data into specific emotion categories for output, thereby obtaining the emotion recognition result of the target object.

[0086] The feature extraction unit is responsible for extracting multimodal emotion features, and can extract feature data from at least two modalities, including visual, audio, and / or text modalities, from multimedia data. By transforming unstructured information such as visual, audio, and text information in multimedia data into machine-understandable structured feature vectors, it captures emotional expression cues from different dimensions.

[0087] The feature fusion unit is responsible for mining the associations between cross-modal features. It is built upon at least one cross-attention mechanism and is used to fuse feature data from at least two modalities pairwise using at least one cross-attention mechanism to obtain at least one fused feature. The cross-attention mechanism is a cross-modal fusion mechanism trained using a contrastive learning strategy. Each cross-attention mechanism is matched with a preset modality combination. For example, if the modality combination matched by a cross-attention mechanism includes visual and audio modalities, then the cross-attention mechanism can perform feature fusion across visual and audio modalities. The feature fusion unit can dynamically calculate the association weights between modalities to capture collaborative emotional patterns between different modalities, thus addressing the problem of one-sided information from a single modality.

[0088] The output unit is responsible for mapping the fused cross-modal features to specific emotion categories, and then outputting the corresponding emotion recognition result. This includes aligning and aggregating multiple fused features output by the feature fusion unit, such as by performing average pooling and then adding them together to obtain the final multimodal fused feature. This multimodal fused feature is then input into a fully connected layer for dimensionality transformation, and the probability distribution of each emotion category is calculated using the Softmax function. The emotion category with the highest probability is selected as the emotion recognition result for the target object.

[0089] By capturing multimodal information from the feature extraction unit, mining cross-modal associations from the feature fusion unit, and accurately classifying the output unit, the emotion recognition model can effectively integrate emotional cues from visual, audio, and textual dimensions, significantly improving the accuracy of recognizing complex emotions.

[0090] The aforementioned emotion recognition method acquires multimedia data of the target object and inputs it into an emotion recognition model to perform emotion recognition, thereby obtaining the emotion recognition result of the target object. The emotion recognition model includes a feature extraction unit, a feature fusion unit, and an output unit. The feature extraction unit extracts feature data from at least two modalities from the multimedia data. The feature fusion unit, constructed based on a cross-attention mechanism, fuses the feature data from at least two modalities pairwise to obtain at least one fused feature. The output unit outputs the corresponding emotion recognition result based on at least one fused feature. After analysis, the inventors and applicants believe that in the field of emotion recognition, multimodal fusion technology based on cross-attention mechanism is a fusion method that extracts text features and then splices and fuses them. However, this splicing and fusion method does not explore the synergistic relationship between different modalities. Since emotion recognition scenarios are not universal and have their own specific characteristics, such as inconsistencies between facial expressions and text / speech expressions, the splicing and fusion methods of related technologies cannot identify these specificities. Therefore, the result of emotion analysis is the result of multimodal collaborative analysis. That is, when the same modal features are combined with features from other modalities, there is a synergistic relationship between the modalities, and the recognized emotion may be different from the emotion recognized by a single modality. If the synergistic relationship between modalities is not explored, errors are likely to occur, such as the misidentification problem of "[visual features are calm expression, text features are surprising copywriting], the recognized emotion should be pleasant, but it is identified as depressed".

[0091] In the context of emotion recognition, this application's embodiments are specifically designed to meet the modal collaboration requirements of emotion recognition. A feature fusion unit built using multiple cross-attention mechanisms is introduced. These mechanisms fuse multimodal features pairwise to deeply explore the collaborative relationships between them, thereby effectively improving emotion recognition accuracy. Therefore, this application's embodiments specifically introduce a feature fusion unit built using cross-attention mechanisms to address emotion analysis needs in non-general scenarios (i.e., emotion recognition scenarios), effectively improving emotion recognition accuracy. Using the emotion recognition method provided in this application's embodiments, in emotion recognition scenarios, the emotion recognition model extracts multimodal features and effectively fuses them through cross-attention mechanisms. By fusing these features, it explores the collaborative relationships between multimodal features, thereby reducing the false positive rate and effectively improving the accuracy of emotion recognition.

[0092] In one exemplary embodiment, reference is made to Figure 2As shown, the feature extraction unit includes at least two of a first feature extraction subunit, a second feature extraction subunit, and a third feature extraction subunit. The process by which the feature extraction unit extracts feature data of at least two modalities from multimedia data may include the following steps 202 to 206. The execution order of steps 202 and 206 is not limited. In one embodiment:

[0093] Step 202: Extract visual features from video frames of multimedia data using the first feature extraction subunit;

[0094] Step 204: Extract audio features from the audio data of the multimedia data through the second feature extraction subunit;

[0095] Step 206: The audio data is converted into text data through the third feature extraction subunit, and text features are extracted from the text data.

[0096] In this embodiment, the feature extraction unit includes at least two of a first feature extraction subunit, a second feature extraction subunit, and a third feature extraction subunit. Each of these feature extraction subunits extracts features from the multimedia data, specifically visual features, audio features, and text features.

[0097] The first feature extraction subunit uses the VIT (Vision Transformer) model to process video frame sequences in multimedia data and extract visual features to capture visual emotional features such as facial expressions and movements. For example, the video in the multimedia data is split into a continuous sequence of video frames (e.g., 10 frames per second). Each frame is input into the VIT model, which uses a self-attention mechanism to model the correlation between pixels within the frame and between frames, capturing facial expression details such as the degree of drooping of the corners of the mouth and the degree of eyelid closure, as well as body movement trends such as curling posture and gesture changes. The output is a feature vector containing visual emotional cues, i.e., the extracted visual features.

[0098] The second feature extraction subunit can use Wav2vec (Waveform-to-Vector, an unsupervised pre-trained model for speech recognition) to extract audio acoustic features, mining emotion-related acoustic features such as intonation and timbre. For example, audio data can be segmented into fixed durations (e.g., every 1.5 seconds) and input into the Wav2vec model. This model uses a pre-trained acoustic encoder to mine emotion-related acoustic features, such as the high energy and rapid speech rate characteristic of anger, and the low fundamental frequency and calm intonation characteristic of sadness, outputting the corresponding audio features.

[0099] The third feature extraction subunit consists of the Whisper model (speech-to-text) and the BERT model, completing the conversion process from audio data to text data and extracting text features from the text data. For example, the Whisper model first converts the audio data into text data, such as converting the speech "This makes me very sad" into text, and then the text data is input into the BERT model. BERT captures the contextual emotional tendency through bidirectional semantic encoding, such as the negative semantic meaning of sadness and the strong emotion of exclamatory sentences, and outputs the corresponding text features.

[0100] The aforementioned feature extraction unit can cover all dimensions of emotional expression, including external behavior, speech rhythm, and language logic, providing high-quality input for subsequent feature fusion and thus effectively improving the accuracy of emotion recognition.

[0101] In an exemplary embodiment, feature data from at least two modalities are paired to obtain n feature data groups. The feature fusion unit includes n cross-attention mechanisms, which are used to fuse the feature data in the n feature data groups, where n is a positive integer.

[0102] In this embodiment, after the feature extraction unit extracts the corresponding feature data, it takes feature data from at least two modalities and groups them together to obtain n feature data groups. The feature fusion unit contains n cross-attention mechanisms corresponding to the above n feature data groups. Each of the n cross-attention mechanisms corresponds one-to-one with one of the n feature data groups and is used to fuse the feature data in the feature data groups.

[0103] For example, the feature extraction unit extracts feature data from m modalities, such as visual, audio, and text modalities. Pairing refers to selecting two modalities from the m modalities and combining them to form feature data groups. The number n can be calculated using a combination formula. For example, if m=3, representing feature data from the visual, audio, and text modalities, then n=3, representing three feature data groups: visual and audio, visual and text, and audio and text. If m=2, representing feature data from the visual and audio modalities, then n=1, representing a feature data group composed of visual and audio modalities.

[0104] This application embodiment uses n cross-attention mechanisms to specifically mine the collaborative relationships between different modalities. By adapting n feature data groups to n cross-attention mechanisms respectively, it can focus on the exclusive association of each modality (such as the matching of textual emotion semantics and visual micro-expressions), avoiding the problems of mutual interference between modalities and dilution of effective information during fusion. This can improve the accuracy of collaborative relationship mining, thereby improving the accuracy and robustness of emotion recognition.

[0105] In an exemplary embodiment, the feature data set includes first modality feature data and second modality feature data. The process of fusing the feature data in the feature data set using a cross-attention mechanism includes:

[0106] Based on the sentiment query intent represented by the first modality feature data, the feature association information between the second modality feature data and the sentiment query intent is determined. Based on the feature association information, the second modality feature data is subjected to feature filtering and fusion targeting the sentiment query intent to obtain the corresponding fused features.

[0107] In this embodiment, the cross-attention mechanism can extract the emotional query intent carried by the first modality feature data, that is, the emotion-related information that the first modality feature data needs to obtain from the second modality feature data. Then, based on the emotional query intent, the feature association information related to the emotional query intent in the second modality feature data can be analyzed. This feature association information is used to quantify the degree of emotional matching between the two. Finally, based on the feature association information, feature information that is highly related to the emotional query intent can be selected from the second modality feature data, and it can be organically integrated with the emotional query intent of the first modality feature data to finally generate a fused feature that simultaneously carries the emotional query intent of the first modality feature data and the core emotional information of the second modality feature data.

[0108] This application embodiment targets the emotional intent of the first modality feature data and only filters information related to the emotional intent in the second modality feature data. This ensures that the fused features can focus on the core emotional representation, accurately capture the emotional synergy between the two modalities, effectively improve the accuracy and robustness of emotion recognition, and is especially suitable for scenarios where single modality feature data is prone to misleading emotion recognition, thereby reducing the false judgment rate of emotions.

[0109] In one exemplary embodiment, reference is made to Figure 3 As shown, based on the sentiment query intent represented by the first modality feature data, the feature association information between the second modality feature data and the sentiment query intent is determined. Then, based on the feature association information, feature filtering and fusion are performed on the second modality feature data for the sentiment query intent to obtain the corresponding fused features. This may include steps 302 to 308, wherein:

[0110] Step 302: Linearly project the first modality feature data onto the query weight matrix to obtain the query feature vector. The query feature vector is used to characterize the sentiment query intent of the first modality feature data.

[0111] Step 304: Linearly project the second modality feature data through the key weight matrix to obtain the key feature vector, which includes feature information for matching sentiment query intent;

[0112] Step 306: Linearly project the second modality feature data using the value weight matrix to obtain the value feature vector, which is the emotion feature used to carry the second modality feature data;

[0113] Step 308: Based on the query feature vector and the key feature vector, generate a feature association weight matrix between the first modality feature data and the second modality feature data, and generate corresponding fused features based on the feature association weight matrix and the value feature vector.

[0114] In this embodiment, the feature data group consists of first-modality feature data and second-modality feature data, such as a feature data group composed of video features and audio features, where video features are first-modality feature data and audio features are second-modality feature data. The cross-attention mechanism is the process of fusing the feature data in this feature data group, which is to model the association between the two modalities through learnable weight matrices. The query weight matrix, key weight matrix, and value weight matrix are all parameters learned during model training. These three weight matrices can map the original feature data to a unified "query" space, "key" space, and "value" space. The "query" space is a mapping space for query feature vectors, used to carry the sentiment query intent of the first-modality feature data; the "key" space is a mapping space for key feature vectors, used to provide index information matching the sentiment query intent; and the "value" space is a mapping space for value feature vectors, used to carry the sentiment information of the second-modality feature data.

[0115] The query weight matrix is ​​responsible for linearly projecting the first modality feature data to extract the emotional intent from the first modality feature data and generate the corresponding query feature vector. This query feature vector is used to characterize the emotional query intent corresponding to the first modality feature data. The emotional query intent characterizes the feature information that needs to be matched from the second modality feature data.

[0116] The key weight matrix is ​​responsible for linearly projecting the second modality feature data to extract feature information from the second modality feature data for matching the sentiment query intent of the first modality feature data, and obtaining the corresponding key feature vector. This key feature vector belongs to the sentiment information index library of the second modality feature data, and the feature information it contains is used to match the sentiment query intent of the first modality feature data.

[0117] The value weight matrix is ​​responsible for linearly projecting the second modality feature data to extract the feature information that plays a key role in emotion recognition from the second modality feature data, and obtain the corresponding value feature vector. This value feature vector can be understood as the carrier of the key feature information of the second modality feature data. That is, the feature information it contains are all features that play a key role in emotion recognition. Through this value feature vector, the interference of irrelevant redundant information on the fused features can be avoided.

[0118] After obtaining the query feature vector, key feature vector, and value feature vector through the above calculations, the similarity matrix between the query feature vector and the key feature vector is calculated. Then, the similarity matrix is ​​normalized using the Softmax function to obtain the feature association weight matrix. This feature association weight matrix is ​​used to quantify the degree of association between each sentiment feature in the second modality feature data and the sentiment query intent of the first modality feature data. Finally, the feature association weight matrix and the value feature vector are multiplied to obtain the corresponding fused features.

[0119] For example, visual features, audio features, and text features are subjected to cross-attention fusion of visual-audio, visual-text, and audio-text features, respectively.

[0120] In one example, targeting visual features and text features After querying the weight matrix respectively Key weight matrix Value weight matrix Linear projection yields the query feature vector Key feature vector Sum eigenvectors The formula is as follows:

[0121]

[0122]

[0123]

[0124] Among them, the projection layer parameters (i.e., the query weight matrix) for different modalities Key weight matrix Value weight matrix Each is set independently, and the shapes of their output tensors are also different. The shape is , and The shape is ), to ensure the expressive power of each modal feature space, among which Let be the vector dimension of each fusion modality feature. The vector dimension of the video features. Let be the vector dimension of the text features. The formula for calculating visual-text cross-attention is as follows:

[0125]

[0126] in, This refers to a fusion of visual and textual features. That is, the key feature vector The transpose of .

[0127] In one example, targeting visual features and audio features After querying the weight matrix respectively Key weight matrix Value weight matrix Linear projection yields the query feature vector Key feature vector Sum eigenvectors The formula is as follows:

[0128]

[0129]

[0130]

[0131] Among them, the projection layer parameters (i.e., the query weight matrix) for different modalities Key weight matrix Value weight matrix Each is set independently, and the shapes of their output tensors are also different. The shape is , and The shape is ), to ensure the expressive power of each modal feature space, among which Let be the vector dimension of each fusion modality feature. The vector dimension of the video features. Let be the vector dimension of the audio features. The formula for calculating the visual-audio cross-attention is as follows:

[0132]

[0133] in, This refers to the fusion of visual and audio features. That is, the key feature vector The transpose of .

[0134] In one example, targeting audio features and text features After querying the weight matrix respectively Key weight matrix Value weight matrix Linear projection yields the query feature vector Key feature vector Sum eigenvectors The formula is as follows:

[0135]

[0136]

[0137]

[0138] Among them, the projection layer parameters (i.e., the query weight matrix) for different modalities Key weight matrix Value weight matrix Each is set independently, and the shapes of their output tensors are also different. The shape is , and The shape is ), to ensure the expressive power of each modal feature space, among which Let be the vector dimension of each fusion modality feature. Let be the vector dimension of the audio features. Let be the vector dimension of the text features. The formula for calculating audio-text cross-attention is as follows:

[0139]

[0140] in, This refers to a fusion of audio and text features. That is, the key feature vector The transpose of .

[0141] Through the above process, the cross-attention mechanism realizes the dynamic association and fusion of the first modality feature data and the second modality feature data, enabling the fused features to accurately capture the emotional coordination pattern between the two modalities. Thus, it can accurately realize emotion recognition based on the fused features and improve the accuracy of emotion recognition results.

[0142] In one exemplary embodiment, reference is made to Figure 4 As shown, the training process of the emotion recognition model may include the following steps 402 to 408, wherein:

[0143] Step 402: Construct a training set. The training set includes at least one training subset. Each training subset corresponds to a cross-attention mechanism. For any cross-attention mechanism, the cross-attention mechanism is used to perform feature fusion on the matched preset modality combination. The preset modality combination includes a first modality and a second modality. The training subset corresponding to the cross-attention mechanism includes positive sample pairs and negative sample pairs. The ground truth sentiments of the two sample data in the positive sample pair are the same, and the ground truth sentiments of the two sample data in the negative sample pair are different. Furthermore, the sentiments represented by the feature data of the two sample data in the first modality are the same, while the sentiments represented by the feature data in the second modality are different.

[0144] In this embodiment, the training set construction logic is highly compatible with the structure of the feature fusion unit when training the emotion recognition model. Since the feature fusion unit contains n cross-attention mechanisms, each responsible for the pairwise fusion of n sets of two-modal features, such as visual-audio, visual-text, and audio-text fusion, the training set needs to simultaneously construct n training subsets. Each training subset is dedicated to serving the corresponding cross-attention mechanism, ensuring that the mechanism can accurately learn the emotion association pattern of the corresponding modality combination.

[0145] Each cross-attention mechanism is matched with a preset modality combination, and the preset modality combinations matched by different cross-attention mechanisms are different. For any cross-attention mechanism, it is assumed that the preset modality combination it fuses includes a first modality and a second modality. That is, the cross-attention mechanism is used to fuse the feature data of the first modality and the feature data of the second modality. For example, if a cross-attention mechanism corresponds to a visual-text modality combination, then the feature data of the first modality is visual feature data and the feature data of the second modality is text feature data. Therefore, the training subset corresponding to this cross-attention mechanism only contains sample data that matches the modality combination, and the sample data is constructed in the form of positive sample pairs and negative sample pairs.

[0146] Positive sample pairs consist of two sample data points labeled with the same ground truth emotion. For example, in the cross-attention mechanism corresponding to visual-audio fusion, a positive sample pair could include sample A and sample B. Sample A corresponds to a ground truth emotion of depression, with visual features representing a droopy face and audio features representing a flat tone of voice. Sample B also corresponds to a ground truth emotion of depression, with visual features representing a unfocused gaze and audio features representing a low volume. These positive sample pairs enhance the model's recognition of feature consistency across different modalities under the same emotion, reducing intra-class discrepancies caused by emotional fluctuations in the voice.

[0147] A negative sample pair consists of two sample data labeled with different ground truth emotions. The first modality features of the two sample data represent the same emotion, while the second modality features represent different emotions. Taking visual-text fusion as an example, a negative sample pair may include sample C and sample D. Among them, the ground truth emotion corresponding to sample C is depressed, the visual feature is represented as a calm expression (corresponding to calm emotion), and the text feature is represented as negative (corresponding to depressed emotion). The ground truth emotion corresponding to sample D is the opposite of the ground truth emotion of sample C. For example, the ground truth emotion of sample D is happy, and the visual feature is represented as a calm expression (corresponding to calm emotion, the same as the emotion represented by the visual feature of sample C), and the text feature is represented as every day is full of surprises (corresponding to happy emotion, different from the emotion represented by the visual feature of sample C). By using these negative sample pairs, the model can focus on the emotional differences between modalities, such as the semantic differences between a calm expression + negative text and a calm expression + positive text. This widens the distance between different emotions in the fusion feature space and specifically addresses the problem of blurred boundaries in the classification of atypical emotions. Atypical emotions refer to emotional expression scenarios where multimodal features are inconsistent and single modal features can easily mislead the judgment of true emotions. That is, scenarios where the emotional tendencies represented by different modalities of the target object (such as text, visual, and speech) are different, and the true emotions cannot be restored by interpreting single modal features in isolation.

[0148] In this embodiment of the application, there is no specific limitation on the number of samples in the training subset. The ratio of positive and negative sample pairs is balanced. For example, the ratio of positive sample pairs to negative sample pairs is 1:1 or 1:2, and the sample data of each training subset covers all target emotion categories.

[0149] Step 404: Perform emotion recognition on each sample data using the initial emotion recognition model to obtain the emotion recognition results corresponding to each sample data, as well as the fusion features output by each cross-attention mechanism for each sample data.

[0150] In this embodiment, after constructing the training set, the sample data in the training set can be input into the initial emotion recognition model for emotion recognition. The output includes the corresponding emotion recognition result and the fusion features output by each cross-attention mechanism for each sample data. The initial emotion recognition model is a basic model with a feature extraction unit, a feature fusion unit, and an output unit, and its parameters are randomly initialized. The training process of the initial emotion recognition model based on the training set involves iteratively optimizing the model parameters through loss feedback to improve cross-modal emotion recognition capabilities.

[0151] Step 406: Train the initial emotion recognition model based on the emotion recognition results, the fusion features output by each cross-attention mechanism for the two sample data in each positive sample pair, and the fusion features output by each cross-attention mechanism for the two sample data in each negative sample pair.

[0152] In this embodiment of the application, each cross-attention mechanism is pre-set with a corresponding training subset, which includes positive sample pairs and negative sample pairs. The fusion features output by each cross-attention mechanism for each sample data can be selected from the fusion features output by each cross-attention mechanism for each sample data in the form of positive sample pairs and negative sample pairs.

[0153] Furthermore, the recognition loss can be calculated based on the fusion features output by each cross-attention mechanism for the two sample data in the positive sample pair, the fusion features output by each cross-attention mechanism for the two sample data in the negative sample pair, and the final emotion recognition result. Based on the recognition loss, comparative learning of the initial emotion recognition model can be initiated.

[0154] Step 408: When the preset training stopping condition is met, the model training is completed.

[0155] During the iterative training of the initial emotion recognition model, if the accuracy of emotion recognition obtained in the current iteration meets the stopping condition, that is, the accuracy of the current emotion recognition model reaches the accuracy threshold (or the model loss is less than the loss threshold), then model training can be stopped, and the model obtained in the current training can be used as the final emotion recognition model. For example, the preset accuracy threshold of the emotion recognition model is 92%. During the iterative training of the initial emotion recognition model, after each training round, the emotion recognition accuracy of the model on the validation set is calculated: Suppose that when training reaches the 75th round, the model's emotion recognition accuracy on the validation set samples reaches 93.2%, which meets the stopping condition of accuracy ≥ accuracy threshold 92%. At this time, model training is stopped, and the model obtained in the 75th training round is used as the final emotion recognition model.

[0156] Alternatively, if the preset number of training iterations has been reached, model training can be stopped, and the currently trained model can be used as the final emotion recognition model. For example, the preset maximum number of model training iterations is 100 rounds. During iterative training, assuming that the model's validation set accuracy has not yet reached the preset accuracy threshold of 92% in the 100th round, but the number of iterations has met the stopping condition of reaching the preset number of 100 rounds, model training is stopped at this point, and the model obtained in the 100th round of training is used as the final emotion recognition model.

[0157] In this embodiment, a training subset is constructed according to the cross-attention mechanism. Positive sample pairs are used to strengthen the consistency of cross-modal features under the same ground truth emotion, while negative sample pairs are used to widen the distance between positive and negative samples in the feature space. This allows each cross-attention mechanism to learn the emotion association pattern of the corresponding modal combination. Finally, through training, the model can not only deeply explore cross-modal emotion collaboration clues, but also effectively solve the problem of fuzzy classification of atypical emotions, thus significantly improving the accuracy of emotion recognition.

[0158] In one exemplary embodiment, reference is made to Figure 5 As shown, in step 406, training the initial emotion recognition model based on the emotion recognition results, the fusion features output by each cross-attention mechanism for the two sample data in each positive sample pair, and the fusion features output by each cross-attention mechanism for the two sample data in each negative sample pair may include the following steps 502 to 504, wherein:

[0159] Step 502: Based on the emotion recognition results, the fusion features output by each cross-attention mechanism for the two sample data in each positive sample pair, the fusion features output by each cross-attention mechanism for the two sample data in each negative sample pair, and the joint loss function, determine the total recognition loss of the initial emotion recognition model. The joint loss function is a loss function constructed with the training objective of simultaneously optimizing the emotion recognition accuracy and the feature fusion discrimination of the cross-attention mechanism.

[0160] Step 504: Adjust the parameters of the initial emotion recognition model based on the total recognition loss.

[0161] In this embodiment, model error can be quantified using a joint loss function, achieving simultaneous optimization of emotion recognition accuracy and cross-modal feature fusion discriminativeness. Ultimately, the emotion recognition performance of the model is improved through parameter adjustment. This joint loss function is a loss function constructed based on a contrastive learning strategy. On one hand, it calculates the emotion recognition loss by comparing the emotion recognition result with the ground truth emotion labeled in the sample data. On the other hand, it calculates the feature fusion loss of each cross-attention mechanism by calculating the similarity of the fusion features of two sample data in a positive sample pair and the difference of the fusion features of two sample data in a negative sample pair. Finally, the emotion recognition loss and the feature fusion loss of each cross-attention mechanism are weighted and summed to obtain the total recognition loss of the initial emotion recognition model.

[0162] After calculating the total recognition loss, all learnable parameters of the initial emotion recognition model can be adjusted using the backpropagation algorithm. These parameters include the query weight matrix, key weight matrix, and value weight matrix of each cross-attention mechanism, as well as the relevant parameters of the feature extraction unit and output unit. The parameter tuning process aims to gradually reduce the total recognition loss, continuously optimizing the model. This includes: optimizing parameters to improve the fit between the predicted results and the true values ​​for the emotion recognition loss; and optimizing parameters to strengthen the correlation between positive samples and fused features, and the discriminative power of negative samples against fused features, ensuring that each cross-attention mechanism can accurately capture the collaborative emotion patterns of the corresponding modality combinations.

[0163] The embodiments of this application construct a joint loss function to deeply bind the model training process with the feature and modality fusion logic of the sample data, so that the training process not only ensures the accuracy of emotion recognition, but also enhances the effectiveness of cross-modal feature fusion, and finally trains a stable emotion recognition model.

[0164] In one exemplary embodiment, reference is made to Figure 6 As shown, in step 502, the total recognition loss of the initial emotion recognition model is determined based on the emotion recognition results, the fusion features output by each cross-attention mechanism for the two sample data in each positive sample pair, the fusion features output by each cross-attention mechanism for the two sample data in each negative sample pair, and the joint loss function. This can include steps 602 to 606, wherein:

[0165] Step 602: Based on the emotion recognition results corresponding to each sample data, determine the emotion recognition loss of the initial emotion recognition model.

[0166] In this embodiment, sample data can be input into an initial emotion recognition model. First, a feature extraction unit extracts corresponding multimodal features. Then, various cross-attention mechanisms fuse these multimodal features, outputting the fused features of the sample data (each cross-attention mechanism outputs corresponding fused features for its respective modality combination). Finally, the fused features are averaged (which can be represented as mean()) and summed to obtain the final multimodal fused features. , By using a fully connected layer and a Softmax function to perform multi-classification on the multimodal fusion features, the emotion recognition result of the sample can be obtained.

[0167] The emotion recognition results of each sample data can be determined using the cross-entropy loss function. Here, the emotion recognition loss is the emotion classification loss, a quantitative indicator that measures the difference between the emotion category predicted by the model and the true emotion value labeled in the sample. The formula for the cross-entropy loss function is shown in formula (1):

[0168] Formula (1)

[0169] Where B is the number of samples in the training subset. Let b be the true label of sample data in category i. Let b be the predicted probability of sample data in category i. For emotion recognition loss.

[0170] Step 604: For any cross-attention mechanism, determine the feature fusion loss of the cross-attention mechanism based on the fusion features of each sample data in the positive sample pairs and the fusion features of each sample data in the negative sample pairs in the training subset corresponding to the cross-attention mechanism.

[0171] Furthermore, a contrastive loss function can be introduced. For any cross-attention mechanism, based on the fusion features of each sample data in the corresponding positive and negative sample pairs, the loss function can be applied to each pair. and Comparative learning, feature fusion loss The result is obtained using the contrastive loss function, where include , as well as , For visual-audio feature fusion loss, For visual-text feature fusion loss, The loss function for audio-text feature fusion is given by formula (2):

[0172] Formula (2)

[0173] in, Characterizing positive sample pairs, and These represent the fusion features corresponding to the two sample data in the positive sample (the fusion features output by the current cross-attention mechanism; taking video-text fusion using the cross-attention mechanism as an example, the positive sample pair includes sample data 1 and sample data 2). Fusion features of sample data 1 ,but Fusion features of sample data 2 ), Characterize sample pairs (including positive and negative sample pairs). and The fusion features corresponding to the two sample data in the sample pair are respectively represented (the fusion features output by the current cross-attention mechanism; taking the cross-attention mechanism for video-text fusion as an example, the sample pair includes sample data 3 and sample data 4). Fusion features of sample data 3 ,but Fusion features of sample data 4 ), The cosine similarity function is used. Let N be the temperature coefficient and N be the training subset. This represents the number of samples in the training subset.

[0174] Step 606: Determine the total recognition loss of the initial emotion recognition model based on the emotion recognition loss and the feature fusion loss of each cross-attention mechanism.

[0175] Furthermore, based on the emotion recognition loss and the feature fusion loss of each cross-attention mechanism, the total recognition loss is calculated. Its calculation formula is as shown in formula (3):

[0176] Formula (3)

[0177] in, The cross-entropy loss represents the emotion classification (i.e., the emotion recognition loss). , , These are the feature fusion losses for visual-audio, visual-text, and audio-text features, respectively.

[0178] Finally, based on the backpropagation algorithm, the gradients of each parameter of the model are calculated according to the total recognition loss. A gradient descent optimizer, such as the Adam optimizer, is used to update all parameters according to the calculated gradient direction. The update magnitude is controlled by the learning rate to reduce the loss. The above process is repeated for multiple rounds of iterative training. After each round of iteration, the model performance is evaluated using a validation set, such as the accuracy of emotion recognition and the convergence of the loss. When the accuracy of the validation set reaches a preset threshold or the loss converges to a stable value and does not decrease significantly for several consecutive rounds, parameter tuning is stopped, and model training is completed.

[0179] The training methods described above not only effectively solve the problems of one-sided information in single modality, insufficient cross-modal fusion, and blurred boundaries of atypical emotion classification, but also significantly enhance the overall accuracy, robustness, and generalization ability of the emotion recognition model.

[0180] In an exemplary embodiment, after performing feature extraction, each feature extraction subunit can use an adaptive filtering mechanism to select high-quality features from the features output by each feature extraction subunit and filter noise, thereby further improving the accuracy of emotion recognition in the emotion recognition model. This includes:

[0181] For each feature vector in the feature information output by the target feature extraction subunit, the corresponding indicator data for each feature vector in each preset indicator dimension of the target modality is determined. The target feature extraction subunit may include a first feature extraction subunit, a second feature extraction subunit, and a third extraction subunit. When the target feature extraction subunit is the first feature extraction subunit, the target modality is a video modality and the feature information is visual features; when the target feature extraction subunit is the second feature extraction subunit, the target modality is an audio modality and the feature information is audio features; when the target feature extraction subunit is the third feature extraction subunit, the target modality is a text modality and the feature information is text features.

[0182] The quality scores of each feature vector are obtained by weighting and summing the corresponding indicator data for each indicator dimension according to the indicator weights for each indicator dimension.

[0183] For any feature vector, if the quality score of the feature vector is less than the dynamic filtering threshold, the feature vector is removed from the feature information to obtain the filtered feature information. The filtered feature information is then filled with gaps to obtain the final target feature information. This final target feature information is then used as a visual feature, audio feature, or text feature to be input into the feature fusion unit for feature fusion.

[0184] In this embodiment, the preset index dimensions for each target modality may include general indexes and modality-specific indexes, thereby ensuring accurate identification of different modal noises.

[0185] For example, the preset metrics dimensions for video modalities may include: feature entropy, frame sharpness score, emotion region confidence score, and inter-frame consistency score. Among them, feature entropy is a general metric used to reflect the concentration of visual features; the lower the entropy value, the more effective it is. The frame sharpness score can be obtained by calculating image sharpness based on Laplacian variance, and is used to quantify the sharpness of the frame. The emotion region confidence score can be obtained by detecting the confidence of key regions of emotional expression such as faces / limbs using the MTCNN (Multi-task convolutional neural network) algorithm. The inter-frame consistency score can be obtained by calculating the mean cosine similarity of features between the current frame and the two adjacent frames, avoiding isolated noisy frames.

[0186] The preset metrics for audio modalities include: feature entropy, signal-to-noise ratio (SNR) score, emotional prosody confidence score, and audio integrity score. Feature entropy is a general metric used to reflect the concentration of acoustic features; the SNR score, calculated using spectral subtraction, is used to filter environmental noise interference; emotional prosody confidence score is obtained by statistically analyzing the matching degree between fundamental frequency fluctuations, energy changes, and known emotional prosodic patterns; and the audio integrity score is obtained by quantifying the proportion of effective speech segments in the audio, eliminating segments with excessively high levels of silence.

[0187] The pre-defined metrics for text modalities include: feature entropy, sentiment confidence, semantic integrity score, and ambiguity score. Feature entropy is a general metric reflecting the concentration of sentiment tendencies among semantic features. Sentiment confidence can be obtained by taking the maximum probability of the sentiment category output by the BERT (Bidirectional Encoder Representations from Transformers) model, used to quantify the clarity of semantic sentiment. Semantic integrity score is calculated based on sentence length and the number of matching sentiment keywords, used to assess the completeness of text semantics. Ambiguity score is obtained by calculating the matching difference between the sentence and the positive / negative sentiment lexicon using a semantic similarity model, used to quantify the degree of semantic ambiguity.

[0188] For example, taking the target modality as the video modality and the feature information as video features, for the visual feature vector of a certain video frame in the video modality, its feature entropy, frame sharpness score, emotion region confidence score, and inter-frame consistency score can be calculated to obtain the complete index data of the feature vector in each index dimension of the video modality.

[0189] After obtaining the indicator data for each indicator dimension, the indicator data can be weighted and summed according to the preset weights of each indicator dimension to finally obtain the quality score of the feature vector (the value ranges from 0 to 1, and the higher the score, the more effective the feature and the less noise).

[0190] The dynamic filtering threshold is an adaptive threshold that adapts to the data distribution, rather than a fixed value. Its determination is divided into a training phase and an inference phase. In the training phase, the mean and standard deviation can be calculated based on the feature quality scores of each batch of sample data, and the dynamic threshold = mean - 0.5 * standard deviation. In the inference phase, the optimal baseline threshold for each modality can be pre-stored based on the feature quality score distribution of the training set, and can be dynamically adjusted according to the overall quality of the input samples. For example, if the signal-to-noise ratio of a certain batch of audio is generally low, the threshold can be lowered by 0.05.

[0191] For each feature vector in the feature information, if its quality score is greater than or equal to the dynamic filtering threshold, it is retained in the filtered feature information; if the quality score is less than the dynamic filtering threshold, it is determined to be a noise feature and removed from the feature information. Since the filtered feature information may have feature sequence gaps (such as the removal of features from 3 consecutive video frames, or the missing features of 2 sentences in a text sequence), a gap completion mechanism is needed to ensure the integrity and temporal correlation of the feature information, so as to avoid affecting the subsequent cross-attention fusion effect.

[0192] For example, for continuously missing feature information, linear interpolation of the effective features before and after can be used to generate complete features to fill the gaps; or, if a feature of a certain modality is missing, the emotional tendency of other modalities can be used to assist in generating complete features to fill the gaps. For example, when text features are missing, a text feature vector with a negative tendency can be generated based on visual (depressed expression) and acoustic (flat tone). Alternatively, for feature information with a missing length greater than a preset length (e.g., 3), a lightweight generation model specific to the target modality can be used to fill the gaps. In this embodiment, no specific limitation is made on the method of gap filling.

[0193] After the gaps are completely filled, high-quality feature information output by each feature extraction submodule is obtained, and the final target feature information is used as visual features, audio features, or text features to be input into the feature fusion unit for feature fusion.

[0194] This application's embodiments evaluate feature quality through multi-dimensional indicators, combine dynamic thresholds to filter multimodal noise, and ensure feature integrity through gap filling. This effectively improves the signal-to-noise ratio and temporal correlation of input features, making the subsequent feature fusion process more accurate, significantly reducing the misjudgment rate of atypical emotions, and enhancing the model's robustness to complex data scenarios.

[0195] To enable those skilled in the art to better understand the embodiments of this application, the embodiments of this application are described below through specific examples.

[0196] Emotion recognition requires the integration of multi-dimensional information, mining correlation clues from behavioral performance, speech features, and semantic content. Video, as a carrier of visual information, records users' facial expressions, body movements, and other behaviors, reflecting outward emotional expressions; audio contains acoustic features such as intonation and rhythm, conveying the phonetic prosody of emotions; and semantic information of user language content can be extracted from the text converted from audio, characterizing emotional states from different modalities. In summary, the visual, acoustic, and semantic features in video all contribute to the recognition of user emotions. This application aims to propose an effective feature extraction and fusion method to enhance the model's ability to recognize emotions.

[0197] First, in the feature extraction stage, VIT is used to process video frame sequences to extract visual feature vectors and capture visual emotional cues such as facial expressions and actions; Wav2vec is used to extract audio acoustic features and mine emotion-related acoustic patterns such as tone and timbre; BERT is used to encode the text after Whisper transformation to obtain semantic features and analyze the emotional tendency of the language content.

[0198] In the feature fusion stage, a cross-attention mechanism is used to achieve pairwise interaction and fusion of multimodal features, and to explore the associations between "visual-acoustic", "visual-semantic", and "acoustic-semantic" features. For example, patients with depression may exhibit a multimodal association pattern of "depressed expression + calm tone + negative text". By calculating the association weights between different modal features through cross-attention, interactive features are fused to comprehensively capture multidimensional emotional associations.

[0199] In the classification output stage, a classification head is constructed based on multimodal fusion features to perform multi-classification of emotions. A fully connected layer combined with a softmax function is used to map the fused features into emotional tendency probabilities, enabling multi-emotional state judgment and improving recognition accuracy by integrating multimodal information.

[0200] Reference Figure 7 The diagram illustrates an emotion recognition method. This embodiment does not impose specific limitations on this method; the emotion recognition process may include stages such as feature extraction, feature fusion, contrastive learning, and model application, wherein:

[0201] In the feature extraction stage, visual features, audio features, and text features can be extracted, including:

[0202] Visual features: video Each frame is input into the VIT model for feature encoding, resulting in... .in, ( Let y be the image of frame x, and y be the total number of frames in the video. is the feature vector extracted from the x-th frame image using the VIT model.

[0203] Audio characteristics: audio Each audio segment is input into the Wav2vec model for feature encoding, resulting in... .in, ( z) represents the x-th audio segment (which can be divided into segments of fixed duration, such as every 3 seconds), and T represents the total number of segments in the audio. is the feature vector extracted by Wav2vec for the x-th audio segment.

[0204] Text features: Convert audio A into text using Whisper. Then it is encoded into a text vector by BERT. Since the converted text from audio A is quite long, it needs to be segmented into sentences. This method uses punctuation marks such as periods and question marks as sentence segmentation symbols to segment the converted text T into sentences. Let x be the x-th sentence following text T. This is the sentence-level vector corresponding to that sentence.

[0205] In the feature fusion stage, cross-attention calculations are performed on the extracted visual features, audio features, and text features for "visual-audio", "visual-text", and "audio-text" respectively. The process is the same as in the aforementioned embodiments, and will not be repeated here in the embodiments of this application.

[0206] In the contrastive learning phase, to enhance the model's sensitivity to intermodal differences and strengthen the classification boundary, a contrastive learning module is introduced after fusing different modal features through cross-attention. This module constructs positive and negative sample pairs. Positive sample pairs are built using modal features from the same emotional video, strengthening the consistency of representations of the same emotional feature and reducing intra-class differences caused by emotional fluctuations. Negative sample pairs are built using modal features from different emotional videos. For example, for "visual-text" fusion features, "normal expression + negative text" and "normal expression + positive text" can be selected as a negative sample pair. Introducing this negative sample pair widens the distance between positive and negative samples in the feature space, solving the problem of blurred classification boundaries for atypical negative emotions that are difficult to express with visual features. A contrastive loss function is introduced to perform contrastive learning on the fusion features of different modalities. Specific details can be found in the descriptions of the aforementioned embodiments, and will not be repeated here.

[0207] Finally, the features of each fused modality are averaged and then summed to obtain the final multimodal fused features. These features are then used for multi-classification through a fully connected layer and a Softmax function, with the cross-entropy loss function being used as the loss function.

[0208] In the model application stage, for unknown videos, the audio file is first separated, and the video and audio are input together into the emotion recognition model, which can directly predict the emotions of the people in the video.

[0209] The emotion recognition method provided in this application adopts a combination of "VIT+Wav2vec+BERT" pre-trained models to extract visual, acoustic, and semantic features respectively. Specifically, it uses a punctuation-based sentence segmentation strategy to process the Whisper-transformed text, which solves the problem of semantic fragmentation in long texts. By relying on the complementarity of visual, acoustic, and semantic features, it comprehensively captures the external manifestations and internal connections of emotions, avoiding the problem of one-sided information from a single modality and effectively improving the accuracy of emotion recognition. A cross-attention module is constructed for pairwise interaction among the three modalities. Features are mapped to query vectors, key vectors, and value vectors through independent linear projection matrices. Combined with Softmax dynamic calculation of association weights, it strengthens modal association mining and improves fusion efficiency. Furthermore, the dynamic allocation of modal weights using the cross-attention mechanism can adapt to different emotional scenarios, comprehensively covering potential connections between modalities and effectively supplementing the basis for emotion judgment, thus further improving the accuracy of emotion recognition. During training, positive sample pairs strengthen the consistency of the same emotional feature, while negative sample pairs widen the distance between classes, accurately distinguishing atypical emotions such as latent depression, which also improves the accuracy of emotion recognition.

[0210] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0211] Based on the same inventive concept, this application also provides an emotion recognition device for implementing the emotion recognition method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more emotion recognition device embodiments provided below can be found in the limitations of the emotion recognition method described above, and will not be repeated here.

[0212] In one exemplary embodiment, such as Figure 8 As shown, an emotion recognition device 800 is provided, including: an acquisition module 802 and a recognition module 804, wherein:

[0213] Module 802 is used to acquire multimedia data of the target object;

[0214] The recognition module 804 is used to input multimedia data into the emotion recognition model to perform emotion recognition and obtain the emotion recognition result of the target object.

[0215] The emotion recognition model includes a feature extraction unit, a feature fusion unit, and an output unit. The feature extraction unit is used to extract feature data of at least two modalities from multimedia data. The feature fusion unit is constructed based on at least one cross-attention mechanism. The feature fusion unit is used to fuse the feature data of at least two modalities pairwise using the at least one cross-attention mechanism to obtain at least one fused feature. The output unit is used to output the corresponding emotion recognition result based on the at least one fused feature. Each cross-attention mechanism is matched with a preset modal combination.

[0216] In an exemplary embodiment, the feature extraction unit includes at least two of a first feature extraction subunit, a second feature extraction subunit, and a third feature extraction subunit. The process by which the feature extraction unit extracts feature data of at least two modalities from the multimedia data includes:

[0217] The first feature extraction subunit extracts visual features from the video frames of the multimedia data;

[0218] The second feature extraction subunit extracts audio features from the audio data of the multimedia data;

[0219] The third feature extraction subunit converts the audio data into text data and extracts text features from the text data.

[0220] In an exemplary embodiment, the feature data of the at least two modalities are paired up to obtain n feature data groups. The feature fusion unit includes n cross-attention mechanisms, which are used to fuse the feature data in the n feature data groups, where n is a positive integer.

[0221] In an exemplary embodiment, the feature data set includes first modality feature data and second modality feature data. The cross-attention mechanism fuses the feature data in the feature data set, comprising:

[0222] Based on the emotion query intent represented by the first modality feature data, the feature association information between the second modality feature data and the emotion query intent is determined, and the second modality feature data is subjected to feature filtering and fusion for the emotion query intent based on the feature association information to obtain the corresponding fused features.

[0223] In an exemplary embodiment, the step of determining feature association information between the second modality feature data and the emotion query intent represented by the first modality feature data, and performing feature filtering and fusion on the second modality feature data based on the feature association information for the emotion query intent to obtain corresponding fused features, includes:

[0224] A query feature vector is obtained by linearly projecting the first modality feature data onto the query weight matrix. The query feature vector is used to characterize the sentiment query intent of the first modality feature data.

[0225] A key feature vector is obtained by linearly projecting the second modality feature data through a key weight matrix. The key feature vector includes feature information for matching the sentiment query intent.

[0226] The second modality feature data is linearly projected onto the value weight matrix to obtain the value feature vector, which is the emotion feature used to carry the second modality feature data;

[0227] Based on the query feature vector and the key feature vector, a feature association weight matrix is ​​generated between the first modality feature data and the second modality feature data, and a corresponding fusion feature is generated based on the feature association weight matrix and the value feature vector.

[0228] In one exemplary embodiment, the training process of the emotion recognition model includes:

[0229] A training set is constructed, which includes at least one training subset. Each training subset corresponds to one cross-attention mechanism. For any cross-attention mechanism, the cross-attention mechanism is used to perform feature fusion on a matched preset modality combination. The preset modality combination includes a first modality and a second modality. The training subset corresponding to the cross-attention mechanism includes positive sample pairs and negative sample pairs. The ground truth sentiments of the two sample data in the positive sample pair are the same, and the ground truth sentiments of the two sample data in the negative sample pair are different. Furthermore, the two sample data represent the same sentiment in the feature data of the first modality, but different sentiments in the feature data of the second modality.

[0230] The initial emotion recognition model is used to perform emotion recognition on each of the sample data to obtain the emotion recognition result corresponding to each of the sample data, as well as the fusion features output by each of the cross-attention mechanisms for each of the sample data;

[0231] The initial emotion recognition model is trained based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, and the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs.

[0232] Model training is completed when the preset training stopping conditions are met.

[0233] In an exemplary embodiment, training the initial emotion recognition model based on the emotion recognition result, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, and the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, includes:

[0234] Based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, and the joint loss function, the total recognition loss of the initial emotion recognition model is determined. The joint loss function is a loss function constructed with the training objective of simultaneously optimizing the emotion recognition accuracy and the feature fusion discrimination of the cross-attention mechanism.

[0235] The initial emotion recognition model is tuned based on the total recognition loss.

[0236] In an exemplary embodiment, determining the total recognition loss of the initial emotion recognition model based on the emotion recognition result, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, and the joint loss function includes:

[0237] Based on the emotion recognition results corresponding to each of the sample data, the emotion recognition loss of the initial emotion recognition model is determined.

[0238] For any of the aforementioned cross-attention mechanisms, the feature fusion loss of the cross-attention mechanism is determined based on the fusion features of each sample data in the positive sample pairs in the training subset corresponding to the cross-attention mechanism and the fusion features of each sample data in the negative sample pairs.

[0239] The total recognition loss of the initial emotion recognition model is determined based on the emotion recognition loss and the feature fusion loss of each of the cross-attention mechanisms.

[0240] The aforementioned emotion recognition device, by extracting multimodal features and fusing them through a cross-attention mechanism, can uncover the synergistic relationships between multimodal features, thereby reducing the misjudgment rate of atypical emotions and effectively improving the accuracy of emotion recognition.

[0241] Each module in the aforementioned emotion recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0242] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an emotion recognition method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0243] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0244] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0245] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0246] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0247] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0248] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0249] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0250] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An emotion recognition method, characterized in that, The method includes: Obtain multimedia data of the target object; The multimedia data is input into the emotion recognition model for emotion recognition, and the emotion recognition result of the target object is obtained. The emotion recognition model includes a feature extraction unit, a feature fusion unit, and an output unit. The feature extraction unit is used to extract feature data of at least two modalities from the multimedia data. The feature fusion unit is constructed based on at least one cross-attention mechanism. The feature fusion unit is used to fuse the feature data of the at least two modalities pairwise using the at least one cross-attention mechanism to obtain at least one fused feature. The output unit is used to output the corresponding emotion recognition result based on the at least one fused feature. Each cross-attention mechanism is matched with a preset modal combination.

2. The method according to claim 1, characterized in that, The feature extraction unit includes at least two of a first feature extraction subunit, a second feature extraction subunit, and a third feature extraction subunit. The process by which the feature extraction unit extracts feature data of at least two modalities from the multimedia data includes: The first feature extraction subunit extracts visual features from the video frames of the multimedia data; The second feature extraction subunit extracts audio features from the audio data of the multimedia data; The third feature extraction subunit converts the audio data into text data and extracts text features from the text data.

3. The method according to claim 1 or 2, characterized in that, The feature data of at least two modalities are paired up to obtain n feature data groups. The feature fusion unit includes n cross-attention mechanisms, which are used to fuse the feature data in the n feature data groups respectively, where n is a positive integer.

4. The method according to claim 3, characterized in that, The feature data set includes first modality feature data and second modality feature data. The cross-attention mechanism fuses the feature data in the feature data set, including: Based on the emotion query intent represented by the first modality feature data, the feature association information between the second modality feature data and the emotion query intent is determined, and the second modality feature data is subjected to feature filtering and fusion for the emotion query intent based on the feature association information to obtain the corresponding fused features.

5. The method according to claim 4, characterized in that, The step involves determining the feature association information between the second modality feature data and the emotion query intent represented by the first modality feature data, and then performing feature filtering and fusion on the second modality feature data based on the feature association information to obtain corresponding fused features, including: A query feature vector is obtained by linearly projecting the first modality feature data onto the query weight matrix. The query feature vector is used to characterize the sentiment query intent of the first modality feature data. The second modality feature data is linearly projected onto the key weight matrix to obtain a key feature vector, which includes feature information for matching the sentiment query intent. The second modality feature data is linearly projected onto the value weight matrix to obtain the value feature vector, which is the emotion feature used to carry the second modality feature data; Based on the query feature vector and the key feature vector, a feature association weight matrix is ​​generated between the first modality feature data and the second modality feature data, and a corresponding fusion feature is generated based on the feature association weight matrix and the value feature vector.

6. The method according to claim 1, characterized in that, The training process of the emotion recognition model includes: A training set is constructed, which includes at least one training subset. Each training subset corresponds to one cross-attention mechanism. For any cross-attention mechanism, the cross-attention mechanism is used to perform feature fusion on a matched preset modality combination. The preset modality combination includes a first modality and a second modality. The training subset corresponding to the cross-attention mechanism includes positive sample pairs and negative sample pairs. The ground truth sentiments of the two sample data in the positive sample pair are the same, and the ground truth sentiments of the two sample data in the negative sample pair are different. Furthermore, the two sample data represent the same sentiment in the feature data of the first modality, but different sentiments in the feature data of the second modality. The initial emotion recognition model is used to perform emotion recognition on each of the sample data to obtain the emotion recognition result corresponding to each of the sample data, as well as the fusion features output by each of the cross-attention mechanisms for each of the sample data; The initial emotion recognition model is trained based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, and the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs. Model training is completed when the preset training stopping conditions are met.

7. The method according to claim 6, characterized in that, The training of the initial emotion recognition model based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, and the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs includes: Based on the emotion recognition results, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, and the joint loss function, the total recognition loss of the initial emotion recognition model is determined. The joint loss function is a loss function constructed with the training objective of simultaneously optimizing the emotion recognition accuracy and the feature fusion discrimination of the cross-attention mechanism. The initial emotion recognition model is tuned based on the total recognition loss.

8. The method according to claim 7, characterized in that, The determination of the total recognition loss of the initial emotion recognition model based on the emotion recognition result, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the positive sample pairs, the fusion features output by each of the cross-attention mechanisms for the two sample data in each of the negative sample pairs, and the joint loss function includes: Based on the emotion recognition results corresponding to each of the sample data, the emotion recognition loss of the initial emotion recognition model is determined; For any of the aforementioned cross-attention mechanisms, the feature fusion loss of the cross-attention mechanism is determined based on the fusion features of each sample data in the positive sample pairs in the training subset corresponding to the cross-attention mechanism and the fusion features of each sample data in the negative sample pairs. The total recognition loss of the initial emotion recognition model is determined based on the emotion recognition loss and the feature fusion loss of each of the cross-attention mechanisms.

9. An emotion recognition device, characterized in that, The device includes: The acquisition module is used to acquire multimedia data of the target object; The recognition module is used to input the multimedia data into the emotion recognition model for emotion recognition and obtain the emotion recognition result of the target object; The emotion recognition model includes a feature extraction unit, a feature fusion unit, and an output unit. The feature extraction unit is used to extract feature data of at least two modalities from the multimedia data. The feature fusion unit is constructed based on at least one cross-attention mechanism. The feature fusion unit is used to fuse the feature data of the at least two modalities pairwise using the at least one cross-attention mechanism to obtain at least one fused feature. The output unit is used to output the corresponding emotion recognition result based on the at least one fused feature. Each cross-attention mechanism is matched with a preset modal combination.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images