Label-feature collaborative learning-based multi-modal emotion recognition method, medium and equipment

Abstract

The invention provides a label-feature collaborative learning-based multi-modal emotion recognition method, a medium and equipment. The method comprises the following steps: carrying out emotion recognition by adopting a multi-mode emotion recognition model; the multi-modal emotion recognition model comprises a data acquisition and preprocessing module, an emotion feature extractor, an adaptive label-feature collaborative learning module, a multi-modal interaction fusion module and an emotion classifier; the data acquisition and preprocessing module acquires and preprocesses behavior signals and physiological signals; an emotion feature extractor extracts single-mode emotion features; the self-adaptive label-feature collaborative learning module generates a single-mode label in a self-adaptive manner and performs single-mode prediction; the multi-modal interaction fusion module carries out feature fusion; and the emotion classifier performs multi-modal prediction to obtain an emotion recognition result. According to the method, accurate semantic alignment from emotion tags to features can be executed to capture discriminative emotion features of each mode, the emotion fuzziness problem is effectively relieved, and the ability of distinguishing different emotions is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and more particularly, to a multi-modal emotion recognition method, medium and device for collaborative learning of labels and features. Background Art

[0002] Human emotions are complex physiological and psychological states affected by thoughts and various sensory inputs. Understanding and accurately recognizing emotions are crucial in many fields, such as the diagnosis and treatment of mental illnesses, virtual reality, and digital twins. Multimodal Emotion Recognition (MER) means integrating multimodal signal data by designing an AI model to imitate the multi-sensory characteristics of human communication, so as to capture the subtle expressions and changes in human true emotions and understand human psychological and emotional factors.

[0003] Spontaneous multimodal emotion data of the human body can be divided into two categories: behavioral and physiological signals, and the recording forms of the two types of signals are mostly time-series data. Among them, behavioral signals include data such as facial expressions, gestures, and speech; physiological signals include data such as electroencephalogram signals, electrocardiogram signals, and respiratory rates. According to theoretical analyses from cognitive neuroscience (A. Moore, I. Gorodnitsky, and J. Pineda, “Eeg mu component responses to viewing emotional faces,” Behavioural brain research, vol. 226, no. 1, pp. 309-316, 2012.) and mathematical statistics (M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, “Analysis of eeg signals and facial expressions for continuous emotion detection,” IEEE Transactions on Affective Computing, vol. 7, no. 1, pp. 17-28, 2015), there is a strong interaction between behavioral and physiological signals; and in the emotion recognition task, the recognition accuracy of combining the two as inputs is higher than that of using only a single type of signal.

[0004] However, due to the high variability of multi-modal sentiment time-series data, the uniformly labeled sentiment tags cannot accurately describe the sentiment semantics expressed by the data at different times, resulting in a mismatch between the tags and the sentiment semantics of the multi-modal data (features), causing the problem of sentiment ambiguity. Specifically, the existing MER dataset regards the induced emotion as the target tag and uses this tag as the only annotation during the entire experiment. However, this coarse-grained annotation is incorrect because the subjects usually only briefly exhibit the induced emotion (such as happiness, anger, etc.) within a few seconds, and the rest of the time is non-emotion-induced. In addition, different senses have different perception speeds for the same emotion, which is reflected in different modalities having their own specific emotion-induced moments (for example, for the same happy emotion, the facial expression modality may appear earlier than the EEG modality). This inconsistency in emotion perception between modalities further exacerbates the problem of sentiment ambiguity.

[0005] The existing technologies (①T.-P. Jung, T. J. Sejnowski et al., “Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing,” IEEE Transactions on Affective Computing, vol. 13, no. 1, pp. 96-107, 2019. ②R. Li, Y. Liang, X. Liu, B. Wang, W. Huang, Z. Cai, Y. Ye, L. Qiu, and J. Pan, “Mindlink-eumpy: an open-source python toolbox for multimodal emotion recognition,” Frontiers in human neuroscience, vol. 15, p. 621493, 2021.) use such uniformly labeled tags to guide the learning of sentiment features for all modalities, indicating that the unified tags also guide the feature learning of data at non-induced moments. This approach hinders the accurate localization of emotion-induced moments by the model in long-duration multi-modal signals, resulting in the sentiment semantics between the annotated tags and multi-modal data samples at any moment becoming blurred. The model's ability to distinguish sentiment features at different times is reduced, further impairing the model's sentiment discrimination ability, and ultimately reflected in the reduction of emotion recognition accuracy. Summary of the Invention

[0006] To overcome the shortcomings and deficiencies in the prior art, the purpose of the present invention is to provide a multi-modal emotion recognition method, medium, and device for label-feature collaborative learning; this method can perform accurate semantic alignment from emotional labels to features to capture discriminative emotional features in each modality, effectively alleviate the problem of emotional ambiguity, and improve the ability to distinguish different emotions.

[0007] To achieve the above purpose, the present invention is realized through the following technical solutions: A multi-modal emotion recognition method for label-feature collaborative learning, which uses a multi-modal emotion recognition model to perform emotion recognition;

[0008] The multi-modal emotion recognition model includes a data collection and preprocessing module, an emotional feature extractor, an adaptive label-feature collaborative learning module, a multi-modal interaction and fusion module, and an emotion classifier;

[0009] Among them, the data collection and preprocessing module is used to collect and preprocess the behavioral signal x 行为 and physiological signal x 生理 of two modalities of data, and align them in time series into a sample pair of multi-modal data;

[0010] There are two emotional feature extractors and two adaptive label-feature collaborative learning modules, which respectively correspond to the behavioral signal x 行为 and physiological signal x 生理 ;

[0011] The two emotional feature extractors respectively extract features from the behavioral signal x 行为 and physiological signal x 生理 in the sample pair to obtain two single-modal emotional features F 单 , where ∈ {behavior, physiology};

[0012] The two adaptive label-feature collaborative learning modules respectively adaptively generate single-modal labels for the single-modal emotional features F 单 obtained by the corresponding emotional feature extractors, and perform predictions on the single-modal emotional features F 单 to obtain single-modal predictions

[0013] The multi-modal interaction and fusion module fuses the single-modal emotional features F 单 obtained by the two emotional feature extractors, and uses a multi-level cross-modal interaction block CMI to enhance the interaction tightness of the single-modal emotional features F 单 to obtain global multi-modal emotional features F 多 ;

[0014] The emotion classifier performs predictions on the global multi-modal emotional features F 多 to obtain multi-modal predictions According to the multi-modal predictions Obtain the emotion recognition result;

[0015] The multi-modal emotion recognition model is a trained multi-modal emotion recognition model; during the training process of the multi-modal emotion recognition model, single-modal prediction and multi-modal prediction are both used as optimization objectives.

[0016] Preferably, during the training process of the multi-modal emotion recognition model, the training samples include behavioral signal x 行为 and physiological signal x 生理 as well as the corresponding multi-modal label y 多 ; Input the training samples into the multi-modal emotion recognition model:

[0017] The data acquisition and preprocessing module obtains the sample pairs of the multi-modal data of each group of training samples;

[0018] The emotion feature extractor obtains the single-modal emotion features F 单 ;

[0019] The adaptive label-feature co-learning module respectively performs three sub-steps of adaptive single-modal label generation, dynamic single-modal label calibration, and single-modal emotion recognition on each group of training samples to obtain the single-modal label y 单 and single-modal prediction The adaptive label-feature co-learning module includes the following sub-steps:

[0020] S1. Adaptive single-modal label generation:

[0021] First, calculate the center vectors C i 高 of the high-emotion categories and the center vectors C i 低 of the low-emotion categories for each modality, where the modalities include the behavioral modality, the physiological modality, and the multi-modal modality; where i ∈ {behavioral, physiological, multi}; then, calculate the Euclidean distances i 高 and i 低 between the data of each modality of the current sample pair and the corresponding center vectors C and Calculate the relative distance value α i ; After that, according to the relative distance value α i , generate the single-modal label y 单 :

[0022] y 单 = y 多 + Δ 多→单

[0023] Among them, Δ 多→单 is the offset value from the multi-modal label y 多 to the single-modal label y 单 and is expressed as:

[0024]

[0025] S2. Dynamic single-modal label calibration: Adopt a dynamic label calibration mechanism to generate single-modal labels;

[0026] S3. Single-modal emotion recognition: Input the single-modal emotion feature F 单 into the corresponding multi-layer perceptron MLP 单 to obtain a single-modal prediction

[0027] The multi-modal interaction and fusion module obtains the global multi-modal emotion feature F 多 of each group of training samples; the emotion classifier obtains the multi-modal prediction 多 of each group of training samples

[0028] Preferably, in the adaptive label-feature co-learning module, in sub-step S1, calculate the central vector C i 高 of the high-emotion category and the central vector C i 低 of the low-emotion category as:

[0029]

[0030] where i ∈ {multi, behavior, physiology}, N is the number of training samples, I(·) is the indicator function, is the general feature of the j-th sample in modality i;

[0031] Calculate the Euclidean distances i 高 and i 低 between the data of each modality of the current sample pair and the corresponding central vectors C and as:

[0032]

[0033] where, is the feature of the current training sample, and d i represents the feature dimension;

[0034] Calculate the relative distance value α i as:

[0035]

[0036] Among them, α i ∈ (-1, 1);

[0037] Sub-step S2, the generation formula of the unimodal label is:

[0038]

[0039] Among them, E represents the number of iterations of the current training, which is set according to the feature length of the unimodal sentiment feature F 单 ; λ changes according to the number of iterations and is calculated as follows:

[0040]

[0041] Sub-step S3, unimodal prediction is:

[0042]

[0043] Preferably, taking the unimodal prediction and the multimodal prediction as the optimization objectives simultaneously means that: the overall loss function is expressed as:

[0044] L 总 = αL 多 + βL 单

[0045] Among them, L 多 is the multimodal loss function; L 单 is the unimodal loss function; α and β are the weights for balancing the multimodal and unimodal losses respectively;

[0046]

[0047] Among them, N is the number of groups of training samples; respectively represent the multimodal label and the unimodal label of the i-th group of training samples; respectively represent the multimodal prediction and the unimodal prediction of the i-th group of training samples; W 单 represents the weight of the unimodal loss:

[0048]

[0049] Among them, softplus(·) represents the Softplus activation function.

[0050] Preferably, in the multimodal interaction and fusion module, each cross-modal interaction block CMI is realized through cross-modal attention; in the calculation of cross-modal attention, the query comes from the target modality t, and the key and value come from the source modality s, that is: set the query Qt = F t W Q , key K s = F s W K , value V s = F s W V ; where W Q , W K , W V are weights respectively; t is one of behavior and physiology, and s is the other; the adaptive interaction of the cross-modal interaction block CMI from the source modality s to the target modality t is:

[0051]

[0052] where d k is the dimension size of the key K s ;

[0053] The cross-modal interaction block CMI of the multi-modal interaction fusion module has two layers; each layer includes cross-modal interaction blocks CMI corresponding to the behavior modality and the physiology modality respectively; the output F t ″ is:

[0054] F t ′ = MHA(LN(F t ), LN(F s )) + F t

[0055] F t ″ = FFN(LN(F t ′)) + F t ′

[0056] where LN(·) is the layer normalization operation; MHA(·) is the multi-head attention; FFN(·) is the feed-forward layer;

[0057] Add the output of the first-layer cross-modal interaction block CMI and element-wise to merge into the hybrid feature F 混合 :

[0058]

[0059] where represents the element-wise addition operation;

[0060] Then, add the output of the second-layer cross-modal interaction block CMI and and the hybrid feature F 混合Input into the Transformer encoder simultaneously to obtain the global multi-modal sentiment feature F 多 :

[0061]

[0062] Among them, Concat(·) represents the feature concatenation operation, and TransEnc(·) represents the Transformer encoder.

[0063] Preferably, the sentiment classifier inputs the global multi-modal sentiment feature F 多 into the multi-layer perceptron MLP 多 , and obtains the multi-modal prediction

[0064]

[0065] In the data acquisition and preprocessing module, the time-series data of the behavioral signal collected is represented as The time-series data of the physiological signal is represented as

[0066] Among them, n and m respectively represent the time-series lengths of the behavioral signal and the physiological signal;

[0067] Preprocess the behavioral signal and the physiological signal respectively: perform object detection and cropping operations on the image data in the behavioral signal; perform downsampling and filtering operations on the physiological signal data; then, perform zero-padding on the time-series data of the behavioral signal and the physiological signal respectively to align the data of each modality in time series; after that, slice the data of each modality into unit samples at a fixed duration of one second, and the sliced behavioral signal and physiological signal form a sample pair of multi-modal data.

[0068] Preferably, the sentiment feature extractor corresponding to the behavioral signal uses the pre-trained convolutional neural network φ 行为 to extract features from the behavioral signal x 行为 :

[0069] F 行为 = φ 行为 (x 行为 );

[0070] The sentiment feature extractor corresponding to the physiological signal uses the one-dimensional convolutional network φ 生理 to extract features from the physiological signal x 生理 ; The one-dimensional convolutional network φ 生理 includes one-dimensional convolutional operations of multiple different scales; each one-dimensional convolutional operation includes a one-dimensional convolutional layer and an activation function layer:

[0071] F 生 phy = φ生 Manage (x 生 Manage).

[0072] A readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the multi-modal emotion recognition method of label-feature collaborative learning as described above.

[0073] A computer device, comprising a processor and a memory for storing programs executable by the processor, and when the processor executes the program stored in the memory, the multi-modal emotion recognition method of label-feature collaborative learning as described above is implemented.

[0074] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0075] 1. The present invention proposes a multi-modal emotion recognition method of label-feature collaborative learning, which can achieve accurate semantic alignment from emotional labels to features, capture discriminative emotional features with clear semantics in each modal data, effectively solve the problem of emotional ambiguity, and improve the emotion discrimination ability of the entire system for difficult-to-recognize samples;

[0076] 2. The present invention designs an adaptive label-feature collaborative learning module to adaptively generate sample-level emotional labels for each modality to accurately locate the emotion induction moments of each modality; in addition, the module includes a dynamic calibration mechanism to comprehensively calibrate the label generation process to improve the emotion consistency between different modalities;

[0077] 3. The present invention designs a multi-modal interaction and fusion module to extract global emotional features rich in emotional information from multi-level multi-modal features to improve the interaction tightness between behavioral and physiological signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a model diagram of the multi-modal emotion recognition method of label-feature collaborative learning of the present invention;

[0079] Figure 2 is a flowchart of the multi-modal emotion recognition method of label-feature collaborative learning of the present invention;

[0080] Figure 3 is a schematic diagram of the principle of the adaptive label-feature collaborative learning module of the present invention;

[0081] Figure 4 is a structural diagram of the cross-modal interaction block of the present invention;

[0082] Figure 5 is a structural diagram of the multi-modal interaction and fusion module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0083] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0084] Embodiment 1

[0085] In this embodiment, a multi-modal emotion recognition method based on label-feature collaborative learning uses a multi-modal emotion recognition model to perform emotion recognition, as shown in Figure 1 and Figure 2 The multi-modal emotion recognition model includes a data acquisition and preprocessing module, an emotional feature extractor, an adaptive label-feature collaborative learning module, a multi-modal interaction and fusion module, and an emotion classifier.

[0086] Among them, the data acquisition and preprocessing module is used to acquire and preprocess the behavioral signal x 行为 and physiological signal x 生理 of two modalities of data, and align them in time series into a sample pair of multi-modal data.

[0087] The time series data of the acquired behavioral signal is expressed as The time series data of the physiological signal is expressed as where n and m respectively represent the time series lengths of the behavioral signal and the physiological signal. In this embodiment, facial expression data is selected as the behavioral signal, and electroencephalogram data is selected as the physiological signal; a camera is used to acquire facial expression data, and a 62-channel electroencephalogram electrode cap distributed according to the international 10-20 system is used to acquire electroencephalogram data. In practical applications, the signal data includes any one or more of data such as facial expressions, gestures, or voices; the physiological signal data includes any one or more of data such as electroencephalogram, electrocardiogram, or electromyogram signals.

[0088] Preprocess the behavioral signal and the physiological signal respectively: perform object detection and cropping operations on the image data in the behavioral signal; perform downsampling and filtering operations on the physiological signal data. For example, for the camera image data, first perform object detection to locate the face, and then crop the image to a size of 64×64 to obtain the behavioral signal of the facial expression data. The original electroencephalogram signal is first downsampled to a sampling rate of 128 Hz; then a band-pass filtering operation from 0.3 to 45 Hz is performed to eliminate noise and remove artifacts, obtaining the physiological signal of the electroencephalogram data.

[0089] Next, perform zero-padding on the time series data of the behavioral signal and the physiological signal respectively to align the data of each modality in time series. Then, slice the data of each modality into unit samples in a non-overlapping manner at a fixed second-level duration. The sliced behavioral signal and physiological signal form a sample pair of multi-modal data.

[0090] There are two emotional feature extractors and two adaptive label-feature collaborative learning modules, which respectively correspond to the behavioral signal x 行为 and the physiological signal x 生理 .

[0091] Two emotion feature extractors respectively extract features from the behavioral signal x 行为 and the physiological signal x 生理 to obtain two single-modal emotion features F 单 , where ∈ {behavior, physiology}.

[0092] For the emotion feature extractor corresponding to the behavioral signal, a pre-trained convolutional neural network φ 行为 (such as ResNet-18 or ResNet-34) is used to extract features from the behavioral signal x 行为 :

[0093] F 行为 = φ 行为 (x 行为 );

[0094] For the emotion feature extractor corresponding to the physiological signal, a one-dimensional convolutional network φ 生理 is used to extract features from the physiological signal x 生理 ; The one-dimensional convolutional network φ 生理 includes multiple one-dimensional convolutional operations of different scales (such as the convolutional kernel size is 1×3 or 1×5, and the stride is 1 or 2); each one-dimensional convolutional operation includes a one-dimensional convolutional layer and an activation function layer:

[0095] F 生 phy = φ 生 phy(x 生 phy).

[0096] Two adaptive label-feature collaborative learning modules respectively adaptively generate sample-level single-modal labels from the single-modal emotion features F 单 obtained by the corresponding emotion feature extractors, as shown in Figure 3 , and make predictions on the single-modal emotion features F 单 to obtain single-modal predictions The adaptive label-feature collaborative learning module includes the following sub-steps:

[0097] S1. Adaptive single-modal label generation:

[0098] Since the single-modal and multi-modal features are distributed in different feature spaces, it is unreasonable to use absolute values to measure the distance between different features. Therefore, the present invention introduces a relative distance value, which is not limited by the difference in feature dimensions. The range of the emotion dimension value is from 1 to 9, and 5 is used here as a threshold to distinguish high-emotion and low-emotion categories. First, calculate the center vector C i 高 of the high-emotion category and the center vector C i 低, the modalities include behavioral modality, physiological modality, and multi-modal; where i ∈ {behavioral, physiological, multi};

[0099]

[0100] where i ∈ {multi, behavioral, physiological}, N is the number of training samples, and I(·) is the indicator function, is the general feature of the j-th sample in modality i;

[0101] Then, calculate the Euclidean distances between the data of each modality of the current sample pair and the corresponding center vectors C i 高 and C i 低 The Euclidean distance between them and

[0102]

[0103] where, is the feature of the current training sample, and d i represents the feature dimension; the present invention measures the relative distances between the modality features and the high and low emotion centers of each sample through the relative distance values, eliminating the influence of dimensional differences.

[0104] Calculate the relative distance value α i :

[0105]

[0106] where α i ∈ (-1, 1); the proposed relative distance value pays more attention to the density and continuity of the relative distances between different modality features and the two emotion centers, realizing the unified and reasonable measurement of different features.

[0107] After that, according to the relative distance value α i , generate the unimodal label y 单 ; Generally speaking, the label difference is related to the distance between the modality feature and the emotion category center, and the unimodal and multimodal emotion semantics are highly correlated. Therefore, the proportional relationship between the unimodal label and the multimodal label and their corresponding relative distance values can be expressed as:

[0108]

[0109] (y 单 - y 多 ) ∝ (α 单 - α 多 ) → y 单 = k2 * [(α 单 - α 多 ) + y多

[0110] Among them, single ∈ {behavior, physiology}, and k1 and k2 are proportionality coefficients. Considering the above two equations comprehensively here, the above equations are integrated into the formula definition of single-modal label generation as follows:

[0111]

[0112] y 单 = y 多 + Δ 多→单

[0113] Among them, Δ 多→单 is the offset value from the multi-modal label y 多 to the single-modal label y 单 , which is expressed as:

[0114]

[0115] S2. Dynamic single-modal label calibration: Although single-modal labels are generated, due to the unstable phenomenon of the continuously updated emotional features during the training process, the label generation processes of different collaborative modules will be unbalanced, resulting in impaired emotional consistency between modalities. For this reason, the present invention proposes a dynamic label calibration mechanism, which considers the emotional information in the current and historical iteration rounds to stabilize the label generation processes of different modalities and enhance emotional consistency. The formula for stabilizing the single-modal label generation process is:

[0116]

[0117] Among them, E represents the number of iterations of the current training, which is set according to the feature length of the single-modal emotional feature F 单 ; λ changes according to the number of iterations and is calculated as follows:

[0118]

[0119] Assuming that the total number of iterations is N, the weights of the single-modal labels follow This means that the weights of the single-modal labels generated later exceed those of the former, which is consistent with the empirical observation, that is, the finally generated stable single-modal label is expressed as the cumulative sum of all the previous ones.

[0120] S3. Single-modal emotion recognition: In order to capture the discriminative emotion features of each modality, it is necessary to maximize the consistency between the generated emotion labels and the predicted labels. For this reason, the single-modal emotional feature F 单 is input into the corresponding multi-layer perceptron MLP 单 to obtain the single-modal prediction

[0121] ​

[0122] The multi-modal interaction fusion module fuses the single-modal emotion features F obtained by the two emotion feature extractors 单 and uses a multi-level cross-modal interaction block CMI to enhance the interaction tightness of the single-modal emotion feature F 单 to obtain a global multi-modal emotion feature F with rich emotion information and high discriminability 多 .

[0123] In the multi-modal interaction fusion module, each cross-modal interaction block CMI is implemented through cross-modal attention; in the cross-modal attention calculation, the query comes from the target modality t, and the key and value come from the source modality s, that is: the query Q is set t = F t W Q , the key K s = F s W K , and the value V s = F s W V ; where W Q , W K , and W V are weights respectively; t is one of behavior and physiology, and s is the other; the adaptive interaction of the cross-modal interaction block CMI from the source modality s to the target modality t is:

[0124]

[0125] where d k is the dimension size of the key K s ;

[0126] The cross-modal interaction block CMI of the multi-modal interaction fusion module has two layers; each layer includes cross-modal interaction blocks CMI corresponding to the behavior modality and the physiological modality respectively, as shown in Figure 4 and Figure 5 ; the output F t ″ of each cross-modal interaction block CMI is:

[0127] F t ′ = MHA(LN(F t ), LN(F s )) + F t

[0128] F t ″ = FFN(LN(F t ′)) + F t ′

[0129] where LN(·) is the layer normalization operation; MHA(·) is the multi-head attention; FFN(·) is the feed-forward layer;

[0130] The output of the first - layer cross - modal interaction block CMI and are element - wise added and merged into the hybrid feature F 混合 :

[0131]

[0132] wherein, represents the element - wise addition operation;

[0133] Then, the output of the second - layer cross - modal interaction block CMI and as well as the hybrid feature F 混合 are simultaneously input into the Transformer encoder to obtain the global multi - modal sentiment feature F 多 :

[0134]

[0135] wherein, Concat(·) represents the feature concatenation operation, and TransEnc(·) represents the Transformer encoder.

[0136] The sentiment classifier makes a prediction on the global multi - modal sentiment feature F 多 to obtain the multi - modal prediction

[0137] In the emotion recognition task, according to the multi - modal prediction the emotion recognition result is obtained.

[0138] The multi - modal emotion recognition model is a trained multi - modal emotion recognition model. During the training process of the multi - modal emotion recognition model, the training samples include behavioral signals x 行为 and physiological signals x 生理 as well as the corresponding multi - modal label y 多 ; The training samples are input into the multi - modal emotion recognition model:

[0139] The data acquisition and pre - processing module obtains the sample pairs of the multi - modal data of each group of training samples;

[0140] The emotion feature extractor obtains the single - modal emotion feature F 单 of each group of training samples;

[0141] The adaptive label - feature co - learning module respectively performs three sub - steps of adaptive single - modal label generation, dynamic single - modal label calibration, and single - modal emotion recognition on each group of training samples to obtain the single - modal label y 单 and the single - modal prediction

[0142] The multi-modal interaction fusion module obtains the global multi-modal sentiment feature F of each group of training samples 多 ; The sentiment classifier obtains the multi-modal prediction of each group of training samples

[0143] Taking the unimodal prediction and the multi-modal prediction as the optimization objectives simultaneously, the overall loss function is expressed as:

[0144] L 总 =αL 多 +βL 单

[0145] where, L 多 is the multi-modal loss function; L 单 is the unimodal loss function; α and β are the weights for balancing the multi-modal and unimodal losses respectively;

[0146]

[0147] where, N is the number of groups of training samples; respectively represent the multi-modal label and the unimodal label of the i-th group of training samples; respectively represent the multi-modal prediction and the unimodal prediction of the i-th group of training samples; W 单 represents the weight of the unimodal loss. In order to make the model pay more attention to the samples with larger emotional differences, the present invention measures the weight of the unimodal loss through the difference between the multi-modal and unimodal labels, and the calculation is as follows:

[0148]

[0149] where, softplus(·) represents the Softplus activation function.

[0150] As an embodiment, experiments were conducted on two mainstream multi-modal emotion datasets (DEAP and MAHNOB-HCI) to verify the effectiveness of the multi-modal emotion recognition method of the present invention. Among them, the DEAP dataset includes modalities such as facial expressions, electroencephalogram (EEG) signals, and respiratory frequency. It includes 32 subjects watching 40 music videos corresponding to different emotions, while recording their physiological data and facial videos of the first 22 subjects. Each video lasts for 1 minute and starts with a 3-second pre-trial baseline. After each trial, the subjects were required to evaluate their emotional responses through questionnaires on valence, arousal, preference, and dominance, with continuous ratings on each dimension ranging from 1 to 9. The MAHNOB-HCI dataset also records behavioral signals such as facial expressions, audio, etc., and physiological signals such as EEG signals, electrocardiogram (ECG) signals, etc. 30 subjects will watch 20 different movie clips, with the duration of each movie clip ranging from 34.9 seconds to 117 seconds, and having 30-second pre-trial and post-trial baselines. After each trial, the subjects need to self-evaluate their emotional states in terms of valence, arousal, and dominance using integers from 1 to 9.

[0151] In terms of the experimental results, a multi-modal emotion recognition method of label-feature collaborative learning proposed by the present invention has the highest accuracy compared with other methods on the DEAP dataset, reaching 98.63% and 98.18% respectively in terms of valence and arousal. On the MAHNOB-HCI dataset, under the individual-dependent experimental setting, this method reaches accuracies of 96.74% and 96.38% respectively, which are the highest values among all the comparison methods. The excellent experimental results on different datasets illustrate the effectiveness of this method in effectively extracting and fusing multi-modal emotion features, as well as the robustness in cross-dataset emotion recognition.

[0152] Embodiment 2

[0153] In this embodiment, a readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the multi-modal emotion recognition method of label-feature collaborative learning described in Embodiment 1.

[0154] Embodiment 3

[0155] In this embodiment, a computer device includes a processor and a memory for storing programs executable by the processor. When the processor executes the program stored in the memory, it implements the multi-modal emotion recognition method of label-feature collaborative learning described in Embodiment 1.

[0156] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A multi-modal emotion recognition method for label-feature collaborative learning, characterized in that: Use a multi-modal emotion recognition model to perform emotion recognition; The multi-modal emotion recognition model includes a data collection and preprocessing module, an emotional feature extractor, an adaptive label-feature co-learning module, a multi-modal interaction and fusion module, and an emotion classifier; Among them, the data acquisition and preprocessing module is used to acquire and preprocess the behavioral signal x of the subject 行为 and the physiological signal x 生理 of the two modalities of data, and align them in time series into a sample pair of multi-modal data; There are two emotion feature extractors and adaptive label-feature co-learning modules, respectively corresponding to the behavior signal x 行为 and the physiological signal x 生理 ; Two emotion feature extractors respectively extract features from the behavior signal x 行为 and the physiological signal x 生理 in the sample pair, and obtain two single-modal emotion features F 单 , where ∈ {behavior, physiology}; Two adaptive label-feature collaborative learning modules respectively generate single-modal labels adaptively from the single-modal sentiment features F obtained by the corresponding sentiment feature extractors, and perform predictions on the single-modal sentiment features F 单 to obtain single-modal predictions 单 ​ The multi-modal interaction fusion module fuses the single-modal emotion features F obtained by the two emotion feature extractors 单 and uses a multi-level cross-modal interaction block CMI to enhance the interaction tightness of the single-modal emotion features F 单 to obtain the global multi-modal emotion feature F 多 ; The sentiment classifier makes a prediction on the global multi-modal sentiment feature F 多 to obtain a multi-modal prediction Based on the multi-modal prediction the emotion recognition result is obtained; The multimodal emotion recognition model is a trained multimodal emotion recognition model; during the training process of the multimodal emotion recognition model, single-modal prediction and multimodal prediction are used as optimization objectives simultaneously.

2. The multi-modal emotion recognition method based on label-feature collaborative learning according to claim 1, wherein: During the training process of the multi-modal emotion recognition model, the training samples include behavioral signals x 行为 and physiological signals x 生理 as well as the corresponding multi-modal label y 多 ; Input the training samples into the multi-modal emotion recognition model: The data collection and preprocessing module obtains sample pairs of multi-modal data for each group of training samples; The emotion feature extractor obtains the unimodal emotion feature F of each group of training samples 单 ; The adaptive label-feature collaborative learning module performs three sub-steps of adaptive unimodal label generation, dynamic unimodal label calibration, and unimodal emotion recognition on each group of training samples, respectively, to obtain the unimodal label y of each group of training samples 单 and unimodal prediction The adaptive label-feature collaborative learning module includes the following sub-steps: S1. Adaptive single-modal label generation: First, calculate the central vectors C of the high-emotion categories for each modality i 高 and the central vectors C of the low-emotion categories i 低 , where the modalities include the behavioral modality, the physiological modality, and the multi-modal modality; among them, i ∈ {behavioral, physiological, multi}; then, calculate the Euclidean distances between the data of each modality of the current sample pair and the corresponding central vectors C i 高 and C i 低 ; and Calculate the relative distance value α i ; after that, according to the relative distance value α i , generate the single-modal label y 单 : y 单 = y 多 + Δ 多→单 where, Δ 多→单 is the offset value from the multi-modal label y 多 to the single-modal label y 单 and is expressed as: S2. Dynamic single-modal label calibration: Use a dynamic label calibration mechanism to generate single-modal labels; S3. Single-modal emotion recognition: Input the single-modal emotion feature F 单 into the corresponding multi-layer perceptron MLP 单 to obtain the single-modal prediction Global multi-modal emotion feature F of the multi-modal interaction fusion module 多 Obtain the global multi-modal emotion feature F of each group of training samples 多 ; The emotion classifier obtains the multi-modal prediction of each group of training samples 3. The multi-modal emotion recognition method based on label-feature collaborative learning according to claim 2, characterized in that: The adaptive label-feature collaborative learning module, sub-step S1, calculates the central vectors C of the high-emotion categories of each modality i 高 and the central vectors C of the low-emotion categories i 低 as follows: where \(i\in\{\text{multi},\text{behavior},\text{physiology}\}\), \(N\) is the number of training samples, \(I(\cdot)\) is the indicator function, is the general feature of the \(j\)-th sample in modality \(i\); Calculate the Euclidean distances between the data of each modality of the current sample pair and the corresponding central vectors C i 高 and C i 低 as follows and are Among them, is the feature of the current training sample, and d i represents the feature dimension; Calculate the relative distance value α i It is: where α i ∈ (-1, 1); In sub-step S2, the generation formula of the single-modal label is: where E represents the current iteration number of training, and is set according to the feature length of the unimodal sentiment feature F 单 ; λ changes according to the iteration number and is calculated as follows: Sub-step S3, unimodal prediction is as follows:

4. The multi-modal emotion recognition method based on label-feature collaborative learning according to claim 2, characterized in that: Taking single-modal prediction and multi-modal prediction as the optimization objectives simultaneously means that the overall loss function is expressed as: L 总 = αL 多 + βL 单 Among them, L 多 is the multi-modal loss function; L 单 is the single-modal loss function; α and β are the weights used to balance the multi-modal and single-modal losses respectively; where N is the number of groups of training samples; respectively represent the multi-modal label and the single-modal label of the i-th group of training samples; respectively represent the multi-modal prediction and the single-modal prediction of the i-th group of training samples; W 单 represents the weight of the single-modal loss: where softplus(·) represents the Softplus activation function.

5. The multimodal emotion recognition method based on label-feature collaborative learning according to claim 1, wherein: In the multi-modal interaction fusion module, each cross-modal interaction block CMI is implemented through cross-modal attention; in the calculation of cross-modal attention, the query Q is set t = F t W Q , the key K s = F s W K , and the value V s = F s W V ; Among them, W Q , W K , W V are weights respectively; t is one of behavior and physiology, and s is the other; the adaptive interaction of the cross-modal interaction block CMI from the source modality s to the target modality t is as follows: where d k is the dimension size of the key K s ; The cross-modal interaction block CMI of the multi-modal interaction fusion module has two layers; each layer includes cross-modal interaction blocks CMI corresponding to the behavioral modality and the physiological modality respectively; the output F t ″ is: F t ′ = MHA(LN(F t ), LN(F s )) + F t F t ″ = FFN(LN(F t ′)) + F t ′ where LN(·) is the layer normalization operation; MHA(·) is the multi-head attention; FFN(·) is the feed-forward layer; The output of the first-layer cross-modal interaction block CMI and are element-wise added and merged into the hybrid feature F 混合 : Among them, represents an element-wise addition operation; Then, the output of the second-layer cross-modal interaction block CMI and along with the hybrid feature F 混合 are simultaneously input into the Transformer encoder to obtain the global multi-modal sentiment feature F 多 : where Concat(·) represents the feature concatenation operation, and TransEnc(·) represents the Transformer encoder.

6. The multi-modal emotion recognition method based on label-feature collaborative learning according to claim 1, wherein: The emotion classifier inputs the global multi-modal emotion feature F 多 into the multi-layer perceptron MLP 多 to obtain the multi-modal prediction 7. The multimodal emotion recognition method based on label-feature collaborative learning according to claim 1, wherein: In the data acquisition and preprocessing module, the time-series data of the behavior signals is represented as The time-series data of the physiological signals is represented as Where n and m respectively represent the time series lengths of the behavior signal and the physiological signal; Preprocess the behavioral signal and the physiological signal respectively: perform object detection and cropping operations on the image data in the behavioral signal; perform downsampling and filtering operations on the physiological signal data; then, perform zero-padding on the time-series data of the behavioral signal and the physiological signal respectively to align the data of each modality in time series; after that, slice the data of each modality into unit samples at a fixed second-level duration, and the sliced behavioral signal and physiological signal form a sample pair of multi-modal data.

8. The multimodal emotion recognition method based on label-feature collaborative learning according to claim 1, characterized in that: The emotion feature extractor corresponding to the behavior signal uses a pre-trained convolutional neural network φ 行为 For the behavior signal x 行为 Perform feature extraction: F 行为 = φ 行为 (x 行为 ); The emotional feature extractor corresponding to the physiological signal uses a one-dimensional convolutional network φ 生理 to extract features from the physiological signal x 生理 ; the one-dimensional convolutional network φ 生理 includes one-dimensional convolutional operations of multiple different scales; each one-dimensional convolutional operation includes a one-dimensional convolutional layer and an activation function layer: F 生 reason = φ 生 reason(x 生 reason).

9. A readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the label-feature co-learning multi-modal emotion recognition method according to any one of claims 1-8.

10. A computer device, comprising a processor and a memory for storing processor-executable programs, characterized in that, When the processor executes the program stored in the memory, it implements the label-feature co-learning multi-modal emotion recognition method according to any one of claims 1-8.

Images