Multimodal emotion recognition method based on label-feature collaborative learning, medium and device

Abstract

The application provides a multi-modal emotion recognition method based on label-feature collaborative learning, a medium and equipment; the method is: using a multi-modal emotion recognition model for emotion recognition; the multi-modal emotion recognition model includes a data acquisition and preprocessing module, an emotion feature extractor, an adaptive label-feature collaborative learning module, a multi-modal interaction fusion module and an emotion classifier; the data acquisition and preprocessing module acquires and preprocesses behavior signals and physiological signals; the emotion feature extractor extracts single-modal emotion features; the adaptive label-feature collaborative learning module adaptively generates single-modal labels and performs single-modal prediction; the multi-modal interaction fusion module performs feature fusion; and the emotion classifier performs multi-modal prediction to obtain an emotion recognition result. This method can perform precise semantic alignment of emotion labels to features to capture discriminative emotion features of each modality, effectively alleviate the emotion ambiguity problem and improve the ability to distinguish different emotions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, and more particularly, to a label-feature collaborative learning multimodal emotion recognition method, medium and device. BACKGROUND

[0002] Human emotion is a complex physiological and psychological state influenced by thoughts and various sensory inputs. Understanding and accurately recognizing emotions is crucial in many fields, such as mental illness diagnosis and treatment, virtual reality, and digital twins. The meaning of Multimodal Emotion Recognition (MER) is to integrate multi-modal signal data through the design of AI models, simulate the multi-sensory characteristics of human communication, capture the subtle expressions and changes in real human emotions, and understand human psychological emotional factors.

[0003] Human spontaneous multi-modal emotional data can be divided into behavioral and physiological signals, and the recording forms of the two types of signals are mostly time series data. Among them, behavioral signals include facial expressions, gestures, and speech data; physiological signals include electroencephalogram (EEG) signals, electrocardiogram (ECG) signals, and respiratory frequency data. According to cognitive neuroscience (A. Moore, I. Gorodnitsky, and J. Pineda, "Eeg mu component responses to viewing emotional faces," Behavioural brain research, vol. 226, no. 1, pp. 309-316, 2012.) and mathematical statistics (M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, "Analysis of eeg signals and facial expressions for continuous emotion detection," IEEE Transactions on Affective Computing, vol. 7, no. 1, pp. 17-28, 2015) and other theoretical analyses, there is a strong interaction between behavioral and physiological signals; and in the emotion recognition task, the recognition accuracy of combining the two as input is higher than that of using only a single signal.

[0004] However, due to the high variability of multi-modal affective time-series data, the unified labeled emotion label cannot accurately describe the emotional semantics expressed by the data at different times, resulting in a mismatch between the label and the emotional semantics of multi-modal data (features), causing emotional ambiguity. Specifically, existing MER datasets consider induced emotions as target labels and use this label as the only annotation throughout the entire experiment. However, this coarse-grained labeling is incorrect because subjects usually only briefly present induced emotions (such as happiness, anger, etc.) for a few seconds, and the rest of the time is non-emotional induction time. In addition, different senses have different perception speeds for the same emotion, and different modalities have their own specific emotional induction time (for example, for the same happy emotion, the facial expression modality may present earlier than the electroencephalogram modality.) This inconsistency in inter-modal emotional perception further exacerbates the problem of emotional ambiguity.

[0005] Existing technologies (① T.-P. Jung, T. J. Sejnowski et al., “Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing,” IEEE Transactions on Affective Computing, vol. 13, no. 1, pp. 96-107, 2019. ② R. Li, Y. Liang, X. Liu, B. Wang, W. Huang, Z. Cai, Y. Ye, L. Qiu, and J. Pan, “Mindlink-eumpy: an open-source python toolbox for multimodal emotion recognition,” Frontiers in human neuroscience, vol. 15, p. 621493, 2021.) use this unified labeled label to guide the emotional feature learning of all modalities, indicating that the unified label also guides the feature learning of non-induction time data. This approach hinders the accurate positioning of emotional induction time by the model in long-period multi-modal signals, resulting in a mismatch between the labeled label and the emotional semantics of multi-modal data samples at any time. The model's ability to distinguish emotional features at different times is reduced, further compromising the model's emotional discrimination ability, ultimately resulting in a decrease in emotion recognition accuracy. SUMMARY

[0006] In order to overcome the defects and deficiencies in the prior art, the purpose of the present application is to provide a label-feature collaborative learning multi-modal emotion recognition method, medium and equipment; the method can perform precise semantic alignment of emotional labels to features to capture discriminative emotional features of each modality, effectively alleviate the emotional ambiguity problem, and improve the ability to distinguish different emotions.

[0007] In order to achieve the above purpose, the present application is realized by the following technical scheme: a label-feature collaborative learning multi-modal emotion recognition method, which adopts a multi-modal emotion recognition model for emotion recognition;

[0008] The multi-modal emotion recognition model comprises a data acquisition and preprocessing module, an emotional feature extractor, an adaptive label-feature collaborative learning module, a multi-modal interaction fusion module and an emotional classifier;

[0009] The data acquisition and preprocessing module is used to acquire and preprocess the behavior signal x 行为 and the physiological signal x 生理 of the subject, and align them in time sequence into sample pairs of multi-modal data;

[0010] The emotional feature extractor and the adaptive label-feature collaborative learning module are two, and correspond to the behavior signal x 行为 and the physiological signal x 生理 respectively;

[0011] The two emotional feature extractors extract features from the behavior signal x 行为 and the physiological signal x 生理 in the sample pair respectively, obtaining two single-modal emotional features F 单 , single∈{behavior, physiological};

[0012] The two adaptive label-feature collaborative learning modules respectively adaptively generate single-modal labels from the single-modal emotional features F 单 obtained by the corresponding emotional feature extractors, and predict the single-modal emotional features F 单 to obtain single-modal predictions

[0013] The multi-modal interaction fusion module fuses the single-modal emotional features F 单 obtained by the two emotional feature extractors, adopts a multi-level cross-modal interaction block CMI to improve the interaction closeness of the single-modal emotional features F 单 , and obtains global multi-modal emotional features F 多 ;

[0014] The emotional classifier predicts the global multi-modal emotional features F 多 to obtain multi-modal predictions According to the multi-modal predictions Obtain emotion recognition results;

[0015] The multimodal emotion recognition model is a trained multimodal emotion recognition model; during the training process of the multimodal emotion recognition model, single-modal prediction is used. and multimodal prediction It also serves as an optimization target.

[0016] Preferably, during the training process of the multimodal emotion recognition model, the training samples include behavioral signals x. 行为 and physiological signals x 生理 And the corresponding multimodal label y 多 Input the training samples into the multimodal emotion recognition model:

[0017] The data acquisition and preprocessing module obtains sample pairs of multimodal data from each group of training samples;

[0018] The sentiment feature extractor obtains the single-modal sentiment features F of each group of training samples. 单 ;

[0019] The adaptive label-feature co-learning module performs three sub-steps on each group of training samples: adaptive unimodal label generation, dynamic unimodal label calibration, and unimodal emotion recognition, to obtain the unimodal label y for each group of training samples. 单 and single-mode prediction The adaptive label-feature co-learning module includes the following sub-steps:

[0020] S1. Adaptive unimodal label generation:

[0021] First, calculate the center vector C of the high sentiment category for each modality. i 高 and the center vector C of the low emotion category i 低 Modalities include behavioral modalities, physiological modalities, and multimodalities; where i ∈ {behavioral, physiological, multimodal}; then, the modal data of each sample pair and its corresponding center vector C are calculated. i 高 and C i 低 Euclidean distance between and Calculate the relative distance value α i Then, based on the relative distance value α i Generate single-modal label y 单 :

[0022] y 单 =y 多 +Δ 多→单

[0023] Where, Δ 多→单 It is a multimodal tag y 多 To single-modal label y 单 The offset value is represented as:

[0024]

[0025] S2. Dynamic Single-Mode Tag Calibration: A dynamic tag calibration mechanism is used to generate single-mode tags;

[0026] S3. Unimodal Emotion Recognition: This involves integrating unimodal emotion features F... 单 Input to the corresponding multilayer perceptron (MLP) 单 To obtain single-mode prediction

[0027] Multimodal interaction fusion module global multimodal emotion features F 多 The global multimodal sentiment features F of each group of training samples were obtained. 多 The sentiment classifier obtains multimodal predictions for each group of training samples.

[0028] Preferably, in the adaptive label-feature co-learning module, sub-step S1 involves calculating the center vector C of the high sentiment category for each modality. i 高 and the center vector C of the low emotion category i 低 for:

[0029]

[0030] Where i ∈ {multiple, behavioral, physiological}, N is the number of training samples, and I(·) is the indicator function. It is a general feature of the j-th sample in modality i;

[0031] Calculate the modal data and corresponding center vector C of the current sample pair. i 高 and C i 低 Euclidean distance between and for:

[0032]

[0033] in, It is the feature of the current training sample, d i Indicates feature dimension;

[0034] Calculate the relative distance value α i for:

[0035]

[0036] wherein, α i ∈(-1,1);

[0037] Sub-step S2, the generation formula of the single-modal label is:

[0038]

[0039] wherein, E represents the number of iterations of the current training, and the feature length of the single-modal sentiment feature F 单 is set according to the single-modal sentiment feature F

[0040]

[0041] Sub-step S3, the single-modal prediction is:

[0042]

[0043] Preferably, the single-modal prediction and the multi-modal prediction are taken as the optimization target at the same time, which means that the overall loss function is represented as:

[0044] L 总 = αL 多 + βL 单

[0045] wherein, L 多 is the multi-modal loss function; L 单 is the single-modal loss function; α and β are weights for balancing the multi-modal and single-modal losses, respectively.

[0046]

[0047] wherein, N is the number of training sample groups; respectively represent the multi-modal label and the single-modal label of the i-th training sample group; respectively represent the multi-modal prediction and the single-modal prediction of the i-th training sample group; W 单 represents the weight of the single-modal loss:

[0048]

[0049] wherein, softplus(·) represents the Softplus activation function.

[0050] Preferably, in the multi-modal interaction fusion module, each cross-modal interaction block CMI is implemented through cross-modal attention; in the cross-modal attention calculation, the query comes from the target modality t, and the key and the value come from the source modality s, that is, the query Qt =F t W Q , key K s =F s W K , value V s =F s W V ; wherein, W Q , W K , W V are weights respectively; t is one of behavior and physiology, and s is the other; the adaptive interaction of the cross-modal interaction block CMI from the source modality s to the target modality t is:

[0051]

[0052] wherein, d k is the dimension size of the key K s ;

[0053] The cross-modal interaction block CMI of the multi-modal interaction fusion module is two layers; each layer includes a cross-modal interaction block CMI corresponding to the behavior modality and the physiological modality respectively; the output F t " of each cross-modal interaction block CMI is:

[0054] F t '= MHA(LN(F t ), LN(F s ))+F t

[0055] F t "=FFN(LN(F t '))+F t '

[0056] wherein, LN(·) is layer normalization operation; MHA(·) is multi-head attention; FFN(·) is forward propagation layer;

[0057] The outputs F 混合 and F 混合 of the first layer cross-modal interaction block CMI are element-wise added and combined into a mixed feature F 多 :

[0058]

[0059] wherein, represents element-wise addition operation;

[0060] Then, the outputs F 多 and F 多 of the second layer cross-modal interaction block CMI and the mixed feature F 行为 are element-wise added and combined into a final output FThe global multi-modal sentiment feature F is obtained by inputting into a Transformer encoder at the same time 多 :

[0061]

[0062] wherein, Concat(·) represents a feature concatenation operation, and TransEnc(·) represents a Transformer encoder.

[0063] Preferably, the sentiment classifier inputs the global multi-modal sentiment feature F 多 into a multi-layer perceptron MLP 多 to obtain a multi-modal prediction

[0064]

[0065] In the data acquisition and preprocessing module, the time series data of the behavior signal is represented as The time series data of the physiological signal is represented as wherein, n and m respectively represent the time series lengths of the behavior signal and the physiological signal.

[0066] The behavior signal and the physiological signal are respectively preprocessed: the image data in the behavior signal is subjected to target detection and cropping operation; the physiological signal data is subjected to down-sampling and filtering operation; then, the time series data of the behavior signal and the physiological signal are respectively subjected to zero padding, so that the modal data are aligned in time series; thereafter, the modal data are sliced into unit samples according to a fixed time length of seconds; the sliced behavior signal and physiological signal constitute a sample pair of multi-modal data.

[0067] Preferably, the behavior signal corresponding sentiment feature extractor adopts a pre-trained convolutional neural network φ 行为 to perform feature extraction on the behavior signal x 行为 :

[0068] F 行为 = φ 行为 (x 行为 );

[0069] The physiological signal corresponding sentiment feature extractor utilizes a one-dimensional convolutional network φ 生理 to perform feature extraction on the physiological signal x 生理 ; the one-dimensional convolutional network φ 生理 contains multiple one-dimensional convolutional operations of different scales; each one-dimensional convolutional operation includes a one-dimensional convolutional layer and an activation function layer:

[0070] F 生 physiological = φ 生 physiological(x 生).

[0071] A readable storage medium, wherein the storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to perform the label-feature collaborative learning based multi-modal emotion recognition method.

[0072] A computer device, comprising a processor and a memory for storing a program executable by the processor, and the processor, when executing the program stored in the memory, implements the label-feature collaborative learning based multi-modal emotion recognition method.

[0073] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0074] 1. The present application provides a label-feature collaborative learning based multi-modal emotion recognition method, which can realize precise semantic alignment of emotional labels to features, capture discriminative emotional features with clear semantics in each modality data, effectively solve the emotional ambiguity problem, and improve the emotion distinguishing ability of the whole system for difficult to identify samples.

[0075] 2. The present application designs an adaptive label-feature collaborative learning module to adaptively generate sample-level emotional labels for each modality to accurately locate the emotional induction moment of each modality. In addition, the module includes a dynamic calibration mechanism to comprehensively calibrate the label generation process to improve the emotional consistency between different modalities.

[0076] 3. The present application designs a multi-modal interaction fusion module to extract global emotional features rich in emotional information from multi-level multi-modal features to improve the interaction closeness between behavior and physiological signals. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 is the model diagram of the label-feature collaborative learning based multi-modal emotion recognition method of the present application;

[0078] Figure 2 is the flowchart of the label-feature collaborative learning based multi-modal emotion recognition method of the present application;

[0079] Figure 3 is the principle diagram of the adaptive label-feature collaborative learning module of the present application;

[0080] Figure 4 is the structure diagram of the cross-modal interaction block of the present application;

[0081] Figure 5 is the structure diagram of the multi-modal interaction fusion module of the present application. DETAILED DESCRIPTION

[0082] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0083] Embodiment One

[0084] The embodiment is a multi-modal emotion recognition method of label-feature collaborative learning, which adopts a multi-modal emotion recognition model for emotion recognition, as shown in Figure 1 and Figure 2 The multi-modal emotion recognition model includes a data acquisition and preprocessing module, a sentiment feature extractor, an adaptive label-feature collaborative learning module, a multi-modal interaction fusion module, and a sentiment classifier.

[0085] The data acquisition and preprocessing module is used to acquire and preprocess the behavioral signals x 行为 and physiological signals x 生理 of two modalities of the subject, and align them in time sequence to form sample pairs of multi-modal data.

[0086] The time sequence data of the behavioral signals is represented as The time sequence data of the physiological signals is represented as Wherein, n and m represent the time sequence lengths of the behavioral signals and the physiological signals, respectively. In this embodiment, facial expression data is selected as the behavioral signals, and electroencephalogram data is selected as the physiological signals; a camera is used to collect the facial expression data, and a 62-channel EEG cap distributed according to the international 10-20 system is used to collect the electroencephalogram data. In actual application, the signal data includes any one or more of facial expression, gesture, or speech data; the physiological signal data includes any one or more of electroencephalogram, electrocardiogram, or electromyogram data.

[0087] The behavioral signals and the physiological signals are preprocessed respectively: the image data in the behavioral signals is subjected to target detection and cropping operation; the physiological signal data is subjected to downsampling and filtering operation. For example, the camera image data is first subjected to target detection to locate the face, and then the image is cropped to a size of 64x64 to obtain the behavioral signals of the facial expression data. The original electroencephalogram signal is first subjected to downsampling processing to reduce the sampling rate to 128Hz; and then subjected to band-pass filtering operation of 0.3 to 45Hz to eliminate noise and remove artifacts to obtain the physiological signals of the electroencephalogram data.

[0088] Next, the time sequence data of the behavioral signals and the physiological signals are zero-padded to align the modal data in time sequence. Then, the modal data is sliced into unit samples in a non-overlapping manner according to a fixed time length of seconds, and the sliced behavioral signals and physiological signals constitute sample pairs of multi-modal data.

[0089] The sentiment feature extractor and the adaptive label-feature collaborative learning module are two, and correspond to the behavioral signals x 行为 and the physiological signals x 生理 , respectively.

[0090] Two sentiment feature extractors extract behavioral signals x from the sample pairs respectively. 行为 and physiological signals x 生理 Feature extraction yields two unimodal sentiment features F. 单 , single ∈ {behavior, physiology}.

[0091] The emotion feature extractor corresponding to the behavioral signal employs a pre-trained convolutional neural network φ 行为 (such as ResNet-18 or ResNet-34) for behavioral signal x 行为 Perform feature extraction:

[0092] F 行为 =φ 行为 (x 行为 );

[0093] The emotion feature extractor corresponding to the physiological signal utilizes a one-dimensional convolutional network φ 生理 For physiological signal x 生理 Feature extraction is performed; a one-dimensional convolutional network φ 生理 It includes multiple one-dimensional convolutional operations of different scales (e.g., kernel size of 1×3 or 1×5, stride of 1 or 2); each one-dimensional convolutional operation consists of a one-dimensional convolutional layer and an activation function layer:

[0094] F 生 Theory = φ 生 reason (x) 生 reason).

[0095] The two adaptive label-feature co-learning modules respectively process the single-modal sentiment features F obtained from the corresponding sentiment feature extractor. 单 Adaptive generation of sample-level unimodal labels, such as Figure 3 As shown, and for the single-modal emotion feature F 单 Single-mode prediction is obtained by performing prediction. The adaptive label-feature co-learning module includes the following sub-steps:

[0096] S1. Adaptive unimodal label generation:

[0097] Since unimodal and multimodal features are distributed in different feature spaces, using absolute values ​​to measure the distance between different features is unreasonable. Therefore, this invention introduces relative distance values, which are not limited by differences in feature dimensions. The range of the emotion dimension value is 1 to 9, and 5 is used here as a threshold to distinguish between high and low emotion categories. First, the center vector C of the high emotion category for each modality is calculated. i 高 and the center vector C of the low emotion category i 低 Modalities include behavioral modalities, physiological modalities, and multimodalities; where i∈{behavioral, physiological, multimodal};

[0098]

[0099] where i∈{multi,behavior,physiology}, N is the number of training samples, I(·) is an indicator function, is the general feature of the jth sample in the modal i;

[0100] Then, the Euclidean distance between the modal data of the current sample pair and the corresponding center vector C i 高 and C i 低 is calculated and

[0101]

[0102] where, is the feature of the current training sample, d i represents the feature dimension; the present application measures the relative distance between the modal feature and the high and low emotion centers of each sample by the relative distance value, eliminating the influence of dimension difference.

[0103] The relative distance value α i is calculated:

[0104]

[0105] where α i ∈(-1,1); the proposed relative distance value pays more attention to the density and continuity of the relative distance between different modal features and the two emotion centers, and realizes the unified and reasonable measurement of different features.

[0106] Then, according to the relative distance value α i , the single-modal label y 单 is generated; generally, the label difference is related to the distance between the modal feature and the emotion category center, and the single-modal and multi-modal emotion semantics are highly related. Therefore, the proportional relationship between the single-modal label and the multi-modal label and the corresponding relative distance value can be expressed as:

[0107]

[0108] (y 单 -y 多 )∝(α 单 -α 多 )→y 单 =k2*[(α 单 -α 多 )+y 多 ]

[0109] wherein, single ∈ {behavior, physiology}, k1 and k2 are proportional coefficients. Here, the above two equations are comprehensively considered, and the above equation is integrated into the formula definition of single mode label generation as follows:

[0110]

[0111] y 单 =y 多 +Δ 多→单

[0112] wherein, Δ 多→单 is the offset value of the multi-modal label y 多 to the single-modal label y 单 , and is expressed as:

[0113]

[0114] S2, dynamic single-modal label calibration: although the single-modal label is generated, due to the instability of the constantly updated emotional features in the training process, the label generation process of different collaborative modules will be unbalanced, resulting in the damage of the emotional consistency between modes. Therefore, the present application proposes a dynamic label calibration mechanism, which considers the emotional information in the current and historical iteration rounds to stabilize the label generation process of different modes and enhance the emotional consistency. The formula for stabilizing the single-modal label generation process is:

[0115]

[0116] wherein, E represents the iteration number of the current training, and is set according to the feature length of the single-modal emotional feature F 单 ; λ changes according to the iteration number and is calculated as follows:

[0117]

[0118] Suppose the total iteration number is N, and the weight of the single-modal label follows This means that the weight of the single-modal label generated in the later period exceeds that of the former, which is consistent with the empirical observation, that is, the final generated stable single-modal label is expressed as the cumulative sum of all previous periods.

[0119] S3, single-modal emotion recognition: in order to capture the discriminative emotional features of each mode, it is necessary to maximize the consistency between the generated emotional label and the predicted label. Therefore, the single-modal emotional feature F 单 is input into the corresponding multi-layer perceptron MLP 单 to obtain the single-modal prediction

[0120]

[0121] The multimodal interaction fusion module fuses the single-modal sentiment features F 单 The multimodal interaction fusion module fuses the single-modal sentiment features F 单 The multimodal interaction fusion module fuses the single-modal sentiment features F 多 .

[0122] In the multimodal interaction fusion module, each cross-modal interaction block CMI is implemented through cross-modal attention; in the cross-modal attention calculation, the query comes from the target modality t, and the key and the value come from the source modality s, that is, the query Q t t W Q , the key K s s W K , and the value V s s W V ; wherein W Q , W K , and W V are weights; t is one of the behavior and the physiology, and s is the other; the adaptive interaction of the cross-modal interaction block CMI from the source modality s to the target modality t is:

[0123]

[0124] wherein d k is the dimension size of the key K s .

[0125] The cross-modal interaction block CMI of the multimodal interaction fusion module has two layers; each layer includes a cross-modal interaction block CMI corresponding to the behavior modality and the physiological modality, respectively, as shown in Figure 4 and Figure 5 The output F t ″ of each cross-modal interaction block CMI is:

[0126] F t ′=MHA(LN(F t ),LN(F s ))+F t

[0127] F t ″=FFN(LN(F t ′))+F t ′

[0128] wherein LN(·) is layer normalization operation; MHA(·) is multi-head attention; and FFN(·) is a forward propagation layer;

[0129] ​​​The output of the first layer cross-modal interaction block CMI and are element-wise added to obtain the mixed feature F 混合 :

[0130]

[0131] wherein, represents an element-wise addition operation;

[0132] The output of the second layer cross-modal interaction block CMI and and the mixed feature F 混合 are simultaneously input into a Transformer encoder to obtain a global multi-modal sentiment feature F 多 :

[0133]

[0134] wherein, Concat(·) represents a feature concatenation operation, and TransEnc(·) represents a Transformer encoder.

[0135] The sentiment classifier predicts the global multi-modal sentiment feature F 多 to obtain a multi-modal prediction

[0136] In the emotion recognition task, according to the multi-modal prediction an emotion recognition result is obtained.

[0137] The multi-modal emotion recognition model is a trained multi-modal emotion recognition model. In the training process of the multi-modal emotion recognition model, the training samples include behavior signals x 行为 and physiological signals x 生理 and corresponding multi-modal labels y 多 ; the training samples are input into the multi-modal emotion recognition model:

[0138] The data acquisition and preprocessing module obtains sample pairs of multi-modal data of each group of training samples.

[0139] The sentiment feature extractor obtains single-modal sentiment features F 单 of each group of training samples.

[0140] The adaptive label-feature collaborative learning module performs adaptive single-modal label generation, dynamic single-modal label calibration, and single-modal emotion recognition on each group of training samples to obtain single-modal labels y 单 and single-modal predictions

[0141] The multi-modal interaction fusion module obtains global multi-modal sentiment features F of each group of training samples 多 The sentiment classifier obtains multi-modal predictions of each group of training samples

[0142] The single-modal prediction and the multi-modal prediction are simultaneously used as optimization objectives, and the overall loss function is represented as:

[0143] L 总 = aL 多 + bL 单

[0144] Wherein, L 多 is a multi-modal loss function; L 单 is a single-modal loss function; a and b are weights for balancing multi-modal and single-modal losses, respectively;

[0145]

[0146] Wherein, N is the number of groups of training samples; respectively represent multi-modal labels and single-modal labels of the i-th group of training samples; respectively represent multi-modal predictions and single-modal predictions of the i-th group of training samples; W 单 represents the weight of single-modal loss. In order to make the model pay more attention to samples with greater emotional differences, the weight of single-modal loss is measured by the difference between multi-modal and single-modal labels, and is calculated as follows:

[0147]

[0148] Wherein, softplus(·) represents a Softplus activation function.

[0149] As an embodiment, experiments are conducted on two mainstream multi-modal emotion datasets (DEAP and MAHNOB-HCI) to verify the effectiveness of the multi-modal emotion recognition method. The DEAP dataset includes facial expressions, electroencephalogram signals, and breathing rates, and includes 32 subjects watching 40 different emotional music videos, while recording their physiological data and facial videos of the first 22 subjects. Each video lasts 1 minute, and starts with a 3-second pre-test baseline. After each test, the subjects are asked to evaluate their emotional response through a questionnaire of valence, arousal, preference, and dominance, with continuous scores from 1 to 9 for each dimension. The MAHNOB-HCI dataset also records behavioral signals such as facial expressions, audio, and physiological signals such as electroencephalogram signals and electrocardiogram signals. 30 subjects will watch 20 different movie clips, each with a duration of 34.9 seconds to 117 seconds, and a 30-second pre-test and post-test baseline. After each test, the subjects need to self-evaluate their emotional state in terms of valence, arousal, and dominance using integers from 1 to 9.

[0150] In terms of experimental results, the label-feature collaborative learning multi-modal emotion recognition method proposed in the present application has the highest accuracy compared to other methods on the DEAP dataset, reaching 98.63% and 98.18% in valence and arousal, respectively. On the MAHNOB-HCI dataset, the method achieves an accuracy of 96.74% and 96.38% in the individual-dependent experimental setting, which is the highest value among all comparison methods. The excellent experimental results on different datasets demonstrate the effectiveness of the method in extracting and fusing multi-modal emotional features, as well as its robustness in cross-dataset emotion recognition.

[0151] Embodiment Two

[0152] The embodiment is a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the label-feature collaborative learning multi-modal emotion recognition method of embodiment one.

[0153] Embodiment Three

[0154] The embodiment is a computer device, including a processor and a memory for storing a program executable by the processor, and the processor executes the program stored in the memory to implement the label-feature collaborative learning multi-modal emotion recognition method of embodiment one.

[0155] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A multi-modal emotion recognition method of label-feature co-learning, characterized in that: Adopt a multi-modal emotion recognition model to perform emotion recognition; The multi-modal emotion recognition model comprises a data acquisition and preprocessing module, an emotion feature extractor, an adaptive label-feature collaborative learning module, a multi-modal interaction fusion module and an emotion classifier; The data acquisition and preprocessing module is configured to acquire and preprocess the behavior signal x of the subject 行为 and the physiological signal x 生理 of the subject into a sample pair of multimodal data in time sequence. The affective feature extractor and the adaptive label-feature co-learning module are two respectively, and correspond to the behavior signal x 行为 and the physiological signal x 生理 respectively. Two affect feature extractors extract features from the behavioral signals x 行为 and the physiological signals x 生理 in the sample pair, respectively, to obtain two single-modal affect features F 单 , single e {behavior, physiology}. Two adaptive label-feature co-learning modules respectively generate single-modal labels and single-modal features F 单 Adaptive generation of single-modal labels and single-modal features F 单 Prediction of single-modal labels and single-modal features F The multi-modal interaction fusion module fuses the single-modal emotion features F 单 obtained by the two emotion feature extractors to obtain global multi-modal emotion features F 单 by using a multi-level cross-modal interaction block CMI to improve the interaction closeness of the single-modal emotion features F 多 ; The affect classifier is applied to the global multimodal affect feature F 多 A multimodal prediction is made The multimodal prediction is used to determine the affect The affect recognition result is obtained The multi-modal emotion recognition model is a trained multi-modal emotion recognition model; in the training process of the multi-modal emotion recognition model, the single-modal prediction and the multi-modal prediction are simultaneously taken as optimization targets; In the multi-modal interaction fusion module, each cross-modal interaction block CMI is implemented through cross-modal attention; in cross-modal attention calculation, set query Q t =F t W Q , key K s =F s W K , value V s =F s W V ; wherein, W Q , W K , W V are weights; t is one of behavior and physiology, and s is the other; adaptive interaction of the cross-modal interaction block CMI from the source modality s to the target modality t is: wherein d k is the dimension size of the key K s . The cross-modal interaction block CMI of the multi-modal interaction fusion module is two layers; each layer includes a cross-modal interaction block CMI corresponding to the behavior mode and the physiological mode respectively; the output F of each cross-modal interaction block CMI is: t " is: F t ′ = MHA(LN(F t ), LN(F s ))+ F t F t "= FFN(LN(F t ') + F t ' Wherein, LN(·) is a layer normalization operation; MHA(·) is a multi-head attention; FFN(·) is a forward propagation layer. combining the outputs of the first and second layers of cross-modal interaction blocks CMI and combining the outputs of the first and second layers of cross-modal interaction blocks CMI 混合 : wherein denotes an element-wise addition operation; Then, the output of the second layer cross-modal interaction block CMI and and the mixed features F 混合 are simultaneously input into a Transformer encoder to obtain global multi-modal sentiment features F 多 : Wherein, Concat(·) represents a feature concatenation operation, and TransEnc(·) represents a Transformer encoder.

2. The method of claim 1, wherein: In the training process of the multi-modal emotion recognition model, the training sample includes a behavior signal x 行为 and a physiological signal x 生理 , and a corresponding multi-modal label y 多 ; the training sample is input into the multi-modal emotion recognition model: The data acquisition and preprocessing module obtains sample pairs of multi-modal data of each group of training samples; The affective feature extractor obtains the single-modal affective features F of each group of training samples 单 ; The adaptive label-feature collaborative learning module respectively performs adaptive single-modal label generation, dynamic single-modal label calibration and single-modal emotion recognition on each group of training samples to obtain single-modal labels y of each group of training samples 单 and single-modal prediction The adaptive label-feature collaborative learning module includes the following sub-steps: S1, adaptive single-modal label generation: First, a center vector C of a high emotion category of each modality is calculated i 高 and a center vector C of a low emotion category i 低 The modality includes a behavior modality, a physiological modality, and a multi-modality; wherein i∈{behavior, physiological, multi}; then, a Euclidean distance between each modality data of a current sample pair and the corresponding center vector C i 高 and C i 低 is calculated and A relative distance value α is calculated i ; then, according to the relative distance value α i , a single-modality label y is generated 单 ​ y 单 = y 多 + Δ 多→单 where Δ 多→单 is the offset value from the multi-modal label y 多 to the single-modal label y 单 is represented as: S2, dynamic single-modal label calibration: a dynamic label calibration mechanism is adopted to generate a single-modal label; S3, single modality emotion recognition: the single modality emotion features F 单 are input to a corresponding multi-layer perceptron, MLP 单 to obtain a single modality prediction The multi-modal interaction fusion module obtains global multi-modal sentiment features F of each group of training samples 多 ; the sentiment classifier obtains multi-modal predictions of each group of training samples 3. The method of claim 2, wherein: The adaptive label-feature co-learning module, sub-step S1, calculates the center vector C of each modality of high emotional category i 高 and the center vector C of low emotional category i 低 is: where i e {multi, behavior, physiology}, N is the number of training samples, I(·) is the indicator function, is a generic feature of the jth sample in modality i; Compute the Euclidean distance between each modality data of the current sample pair and the corresponding center vector C i 高 and C i 低 between each modality data of the current sample pair and the corresponding center vector C and is: wherein, is a feature of the current training sample, d i denotes the feature dimension; Calculating the relative distance value a i is: wherein a i ∈(-1, 1); Sub-step S2, the generation formula of the single-modal label is: where E denotes the number of iterations of the current training, and the feature length of the single-modal sentiment feature F 单 is set according to λ, which varies according to the number of iterations and is calculated as follows: Sub-step S3, unimodal prediction is:

4. The method of claim 2, wherein: single-modal prediction and multi-modal prediction simultaneously as an optimization objective, which means that the overall loss function is represented as: L 总 = aL 多 + bL 单 wherein L 多 is a multi-modal loss function; L 单 is a single-modal loss function; and a and b are weights for balancing the multi-modal and single-modal losses, respectively. wherein N is the number of training sample groups; respectively represent the multi-modal label and the single-modal label of the i-th training sample group; respectively represent the multi-modal prediction and the single-modal prediction of the i-th training sample group; W 单 represents the weight of the single-modal loss: Wherein, softplus(·) represents a Softplus activation function.

5. The method of claim 1, wherein: said affect classifier, global multimodal affect features F 多 input to a multilayer perceptron MLP 多 , resulting in multimodal predictions 6. The method of claim 1, wherein: The time series data of the behavior signal collected in the data acquisition and preprocessing module is represented as The time series data of the physiological signal is represented as Wherein, n, m respectively represent the time series length of the behavior signal and the physiological signal. The behavior signal and the physiological signal are preprocessed respectively: the image data in the behavior signal is subjected to target detection and cropping operation; the physiological signal data is subjected to down-sampling and filtering operation; then, the time series data of the behavior signal and the physiological signal are subjected to zero padding respectively, so that the modal data are aligned in time series; thereafter, the modal data are sliced into unit samples according to a fixed time length of seconds, and the sliced behavior signal and physiological signal constitute sample pairs of multi-modal data.

7. The method of claim 1, wherein: The emotion feature extractor corresponding to the behavior signal adopts a pre-trained convolutional neural network φ 行为 The behavior signal x 行为 Feature extraction: F 行为 = φ 行为 (x 行为 ); The affective feature extractor corresponding to the physiological signal utilizes a one-dimensional convolution network φ 生理 extracts features from the physiological signal x 生理 extracts features from the physiological signal x 生理 contains multiple one-dimensional convolution operations of different scales; each one-dimensional convolution operation includes a one-dimensional convolution layer and an activation function layer: F 生理 = φ 生理 (x 生理 ).

8. A readable storage medium, characterized by, The storage medium stores a computer program, and the computer program causes the processor to execute the label-feature collaborative learning multi-modal emotion recognition method of any one of claims 1-7 when executed by the processor.

9. A computer device comprising a processor and a memory for storing a processor- executable program, characterized in that, The processor executes the program stored in the memory, and the label-feature collaborative learning multi-modal emotion recognition method of any one of claims 1-7 is realized.

Images