Emotion ground color multi-label recognition method based on facial expression and multi-source physiological signal fusion

Abstract

This invention discloses a multi-label recognition method for emotional background based on the fusion of facial expressions and multi-source physiological signals, belonging to the field of affective computing and multimodal intelligent information processing technology. Based on an artificial intelligence model, this invention preprocesses and temporally aligns synchronously acquired frontal facial videos with physiological signals such as EEG, ECG, and TENS, constructing corresponding feature extraction models to obtain emotion-related features for each modality. A window-level fusion representation is obtained through modality-weighted fusion based on quality vector constraints. This invention models emotional background recognition as a representation problem of complementary coupling between "explicit behavioral cues" and "internal physiological responses." By fusing and modeling these two types of information, it can, to a certain extent, compensate for the representation bias caused by noise, occlusion, or measurement fluctuations in a single modality, thereby more comprehensively depicting an individual's emotional state.

Description

Technical Field

[0001] This invention relates to the field of emotion computing and multimodal intelligent information processing technology, specifically to a multi-label recognition method for emotion background based on feature extraction, time synchronization alignment, and fusion modeling of facial videos and multi-source physiological signals. Background Technology

[0002] With the development of emotion-aware human-computer interaction, digital healthcare, and mental health-related services, the automatic identification and quantitative assessment of individual emotional information has gradually become an important research direction in affective computing and multimodal intelligent information processing. Emotion-related cues can be reflected by both overt behavioral signals and internal physiological responses, such as facial expressions, speech rhythm, and physiological signals like electroencephalography (EEG), electrocardiography (ECG), and electrodermal activity / electrical activity (GSR / EDA), thus providing the possibility for joint modeling of multi-source information.

[0003] Among the aforementioned information channels, facial video-based expression cues are convenient to acquire and have relatively low deployment costs. However, in natural scenes, they are easily affected by factors such as pose changes, lighting conditions, occlusion, and behavioral masking, leading to representational bias and recognition instability. In contrast, physiological signals can reflect the characteristics related to autonomic nervous system and central nervous system activity to a certain extent, but actual acquisition and engineering processing often face problems such as noise and motion artifacts, significant cross-individual differences, inconsistent sampling rates of different devices, and difficulties in multimodal time synchronization and alignment, thus increasing the difficulty of robust fusion and reliable inference.

[0004] In existing technologies, emotion recognition methods typically build models for each single modality. For example, they use convolutional neural networks or Transformer structures to model facial expression sequences, or they employ feature engineering and deep networks to discriminate signals such as EEG / ECG / GSR. To improve generalization ability and noise resistance, multimodal emotion recognition further employs strategies such as feature-level fusion, decision-level fusion, or hybrid fusion to jointly model multi-source features. Taking the publicly available dataset AMIGOOS as an example, it provides synchronously acquired facial videos and various physiological signals (including EEG, ECG, GSR, etc.), and includes emotion dimension annotations (such as pleasure / arousal) and annotation information related to individual differences, providing basic data conditions for multimodal emotion modeling and evaluation.

[0005] However, existing technologies still have at least the following shortcomings:

[0006] (1) Narrow modeling objectives: Most methods focus on the identification of emotional states within a short time window or continuous dimension regression, and lack a unified and reusable modeling framework for the relatively stable emotional trait representation across contexts and time.

[0007] (2) Insufficient engineering availability: Multimodal data often suffers from time asynchrony, sampling rate differences, mode missing and quality fluctuations in actual acquisition. If there is a lack of targeted alignment, quality assessment and robust fusion mechanism, the fusion output is easily unstable and the system's deployability is reduced.

[0008] (3) Incomplete output expression: The existing results are mostly single-label or single-dimensional regression, lacking a unified multi-label output organization method, as well as the corresponding confidence characterization and interpretation support, which makes it difficult to meet the stable decision-making needs of application. Summary of the Invention

[0009] The purpose of this invention is to provide a multi-label recognition method for emotional background based on the fusion of facial expressions and multi-source physiological signals. It aims to use synchronously acquired facial videos and physiological signals to achieve multi-label recognition of relatively stable emotional background of individuals while ensuring multimodal temporal alignment and feature complementarity. This solves the problems of existing methods being susceptible to single-modal noise, large differences across subjects, difficulty in outputting multi-label results, and insufficient stability.

[0010] The technical solution adopted in this invention is as follows: based on an artificial intelligence model, the synchronously acquired frontal face video and physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG) and conductance of skin (GSR) are preprocessed and time-aligned, and corresponding feature extraction models are constructed to obtain emotion-related features of each modality. Window-level fusion representation is obtained through modality weighted fusion based on quality vector constraints.

[0011] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0012] A multi-label recognition method for emotion background based on the fusion of facial expressions and multi-source physiological signals includes the following steps:

[0013] S1: Multimodal data acquisition, label reading, and window index construction: Acquire facial video data and multi-source physiological signal data corresponding to subject u, and read the emotion annotation information corresponding to the subject; construct a set of window indexes to be processed within the acquisition period according to the preset window length T and step size H, which is used to extract window-level synchronization samples from the preprocessed and time-synchronized data; the ground truth value of the emotion background multi-label is generated by the emotion annotation information according to the preset label construction rules;

[0014] S2: Multimodal preprocessing, unified timeline mapping, and window-level synchronous sample extraction: Perform face detection, key point localization, face region alignment, and cropping on face videos to obtain preprocessed face sequences. Denoising filtering, artifact suppression, and amplitude normalization are performed on the multi-source physiological signals to obtain the preprocessed physiological signals. The video modalities are mapped to a unified timeline based on frame-level timestamps or preset frame rates. The physiological modalities are mapped to the unified timeline based on their respective sampling rates through resampling and / or interpolation. The corresponding data segments of each modality within the same time window are extracted using the window index described in S1 to obtain window-level synchronization samples. ;

[0015] S3: Modal Feature Extraction and Quality Vector Construction: Construct a video modal encoder and a physiological submodal encoder respectively, and extract features from the window-level synchronization samples obtained in S2; the video modal encoder takes the face frame sequence within the window as input and outputs a video modal window-level representation; the physiological submodal encoder takes the window-level signal segments of each physiological submodal actually acquired as input and outputs a window-level representation of the corresponding physiological submodal; calculate at least one quality index for the video modality and each physiological submodal, and construct the corresponding modal quality vector q, which is used to generate the fusion weights of each modality;

[0016] S4: Modal weighted fusion, cross-window aggregation, and multi-label prediction of emotional background based on quality vector constraints: Based on the window-level representation of each modality and the corresponding quality vector, the fusion weight of each modality in the current time window is calculated, and the window-level representations of each modality are weighted and fused to obtain the window-level fused representation; at the subject level, cross-window aggregation is performed on the fused representations of multiple time windows to obtain the subject-level emotional background representation; then, the multi-label prediction result of emotional background and its corresponding confidence are output through the multi-label prediction head, and the model parameters are jointly optimized by multi-label loss and stability constraints.

[0017] Furthermore, S1 includes:

[0018] S1-1: The acquired multimodal data is defined as follows:

[0019] The raw video modality data consists of facial video sequences of the subjects during the acquisition period. The multi-source physiological signal data includes one or more of EEG, ECG, and GSR. Each physiological submodal actually acquired is subsequently encoded and fused. The raw physiological modality data consists of EEG, ECG, and GSR signal sequences acquired synchronously by the subjects during the acquisition period. The emotion annotation information includes valence, arousal, control, and / or other emotion-related annotation information.

[0020]

[0021] in, A sequence of face frames aligned within a window. For each physiological signal segment within the window (m∈{EEG,ECG,GSR}); the truth value of the emotional background multi-label is defined as:

[0022]

[0023] Where L is the number of emotional background color labels; the true values ​​of the emotional background color multi-label are generated by the emotional labeling information according to the preset label construction rules. In one embodiment, the corresponding multi-label true value vector is formed by statistically aggregating and threshold mapping the valence, arousal and control of the subject under multiple stimulus tasks.

[0024] S1-2: The window index is constructed as follows: Let the unified time axis be t, the window length be T, and the step size be H. Then the time range of the k-th window is:

[0025]

[0026] Where T is the window length, H is the step size, and tu,0 is the starting timestamp of subject u; the window index is only used to define the time range and is not used for modal preprocessing or fine-grained timing synchronization in this step.

[0027] Furthermore, S2 includes:

[0028] S2-1: Based on the window index described in S1, extract window-level synchronization samples from the preprocessed and time-aligned data. The data of each modal sample within the window are represented as follows:

[0029] Physiological modality preprocessing: Filtering, artifact suppression, standardization, and missing fragment labeling were performed on EEG, ECG, and GSR, respectively;

[0030]

[0031] Video modal preprocessing: Detection, keypoint localization, alignment, cropping, and normalization are performed on video frames to obtain a sequence of face frames within a window.

[0032]

[0033] S2-2: Unified Timeline Mapping and Sample Extraction: Based on the video timestamp or frame rate and the sampling rate of each physiological modality, the data of each modality is mapped to a unified timeline, and window-level synchronous samples are extracted according to the window index described in S1.

[0034] Furthermore, in S3, the output of the modal encoder is defined as: establishing a feature extraction network for each modality. This yields a window-level representation:

[0035] .

[0036] Furthermore, S4 includes:

[0037] S4-1: Quality-perceived fusion can be represented as: First, calculate the fusion weights for each modality:

[0038]

[0039]

[0040] Where g(·) is the weight generation network, and [·;·] denotes vector concatenation; then the window-level fusion representation is obtained:

[0041] ;

[0042] S4-2: Cross-window aggregation of emotional undertone representation is as follows: Let subject u have a total of Each window represents the underlying emotional tone. Defined as:

[0043]

[0044] S4-3: Multi-label prediction and confidence output is: Sentiment background color multi-label prediction vector:

[0045]

[0046] Where σ(·) is the Sigmoid function, and Wb is... For learnable parameters, The values ​​of each dimension can be used as the confidence level of the corresponding label.

[0047] S4-4: The training objective function uses multi-label binary cross-entropy loss, and a stability regularization term can be added:

[0048]

[0049] To improve the cross-window stability of emotional background, a stability regularization term is introduced, defined as:

[0050]

[0051] And the total loss is obtained:

[0052] .

[0053] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0054] This invention models emotional background recognition as a representation problem that complements and couples "explicit behavioral cues" and "internal physiological responses": facial expressions provide easily observable explicit information, while multi-source physiological signals reflect changes in an individual's internal state. By fusing and modeling these two types of information, it can, to some extent, compensate for the representational biases caused by noise, occlusion, or measurement fluctuations in a single modality, thereby providing a more comprehensive characterization of an individual's emotional state.

[0055] This invention describes the extraction of time-stable emotional features as a modeling challenge that combines "intra-window fluctuations" and "cross-window stability." It employs a window-level fusion combined with subject-level aggregation to characterize relatively stable emotional backgrounds across time, reducing the impact of single-measurement errors and noise interference. Simultaneously, it introduces a multi-label output and confidence level representation mechanism, which can simultaneously provide multiple background labels and their corresponding confidence information, facilitating result interpretation and subsequent use, thereby enhancing its applicability in scenarios such as mental health auxiliary assessment, emotion monitoring, and intelligent interaction. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of multimodal data acquisition and annotation based on the AMIGOS dataset.

[0057] Figure 2 This is the overall flowchart of the present invention for multi-label recognition of emotional background based on the fusion of facial expressions and multi-source physiological signals.

[0058] Figure 3 This is a schematic diagram of the sliding window segmentation and multimodal time alignment process of the present invention.

[0059] Figure 4 This is a schematic diagram of the video modal encoder and physiological submodal encoder of the present invention.

[0060] Figure 5 This is a schematic diagram of the modal weighted fusion network based on quality vector constraints of the present invention. Detailed Implementation

[0061] The technical solution of the present invention will be further described and illustrated below with reference to the embodiments.

[0062] Example 1:

[0063] A multi-label method for emotion background recognition based on the fusion of facial expressions and multi-source physiological signals is proposed, implemented on the publicly available dataset AMIGOS. Figures 1-5 As shown, it includes the following steps:

[0064] S1: Multimodal data acquisition, label reading, and window index construction.

[0065] Multimodal raw data and corresponding emotion annotation information of subjects are read from the AMIGOS dataset. The multimodal raw data includes frontal face videos and physiological signals such as EEG, ECG, and GSR. A window index is constructed based on a preset window length and step size. After preprocessing and unified time axis mapping, window-level synchronous sample units are extracted based on the window index.

[0066] The sample and modal representations are defined as follows:

[0067]

[0068] The truth value definition for multi-labeled emotional background is:

[0069]

[0070] Where L is the number of emotional background labels, and the "emotional background" referred to in this invention refers to the relatively stable emotional tendency shown by the same subject across time windows; the stable emotional feature vector obtained by fusing multiple time window representations through a predetermined aggregation rule is called the "emotional background representation".

[0071] S2: Multimodal preprocessing, unified timeline mapping, and window-level synchronous sample extraction

[0072] Preprocessing of face videos includes face detection, key point localization, face region alignment, cropping, and normalization to obtain a preprocessed face frame sequence. Physiological signals such as EEG, ECG, and GSR were subjected to filtering, noise reduction, artifact suppression, amplitude normalization, and missing fragment labeling to obtain preprocessed physiological signals. Let the unified time axis be t, the window length be φ, the step size be H, and the range of the k-th window be:

[0073]

[0074] T: Window length, H: Step size : The start timestamp of subject 𝑢.

[0075] Each modal sample within the window is represented as follows

[0076] Physiological modalities:

[0077]

[0078] Video modality:

[0079]

[0080] S3: Modal feature extraction and quality vector construction.

[0081] A visual encoder and a physiological encoder are constructed separately to extract features from the window-level inputs of the video modality and each physiological submodality. The visual encoder takes a sequence of face frames within a window as input and outputs a window-level representation of the video modality. The physiological encoder takes EEG, ECG, and GSR signal segments within a window as input and outputs a window-level representation of the corresponding physiological submodality. For any modality m, its window-level representation is defined as:

[0082]

[0083] Simultaneously, quality indices are calculated for each mode to form a quality vector. (For example: face detection confidence / occlusion ratio, EEG artifact ratio, ECG R peak detection rate, GSR missing rate, etc.), used for adaptive weighting in subsequent fusion.

[0084] S4: Modal weighted fusion, cross-window aggregation, and multi-label prediction of sentiment background based on quality vector constraints

[0085] Quality-perceived weighted fusion of window-level representations of each modality is performed to obtain window-level fused representations; at the subject level, cross-window aggregation of the fused representations of multiple windows is performed to form emotional background representations; finally, the multi-label emotional background multi-label results and confidence scores are output through a multi-label prediction head, and the model parameters are jointly optimized by multi-label loss and stability constraints during the training phase.

[0086] 1. Multimodal data preprocessing and alignment module

[0087] (1) Face video processing: Perform detection → key point → alignment → cropping → normalization on each frame to obtain the aligned face sequence. It also outputs quality metrics (such as detection confidence and occlusion rate) simultaneously, forming a corresponding quality vector. .

[0088] (2) Physiological signal processing: bandpass filtering, artifact suppression, and Z-score normalization were performed on EEG / ECG / GSR respectively; missing segments were masked and recorded to form a missing mask (or missing indicator vector). This is used for subsequent encoding and fusion.

[0089] (3) Sliding window alignment: Generate synchronous window samples for subsequent encoding and fusion.

[0090] 2. Modal Feature Encoding Module

[0091] This embodiment adopts a parallel structure of "video modal encoder + physiological submodal encoder":

[0092] (1) Visual encoder: The input is a sequence of face frames within the window. The output can be structured using "2D-CNN / ViT to extract frame features + temporal model (Transformer / LSTM / TCN) aggregation". .

[0093] (2) Physiological encoder: EEG / ECG / GSR are encoded using 1D-CNN / TCN or frequency-time hybrid encoding (such as STFT / wavelet + 1D-CNN) respectively. .

[0094] 3. Quality perception fusion and subject-level aggregation module

[0095] (1) Quality perception weight calculation: First, generate fusion weights based on modal features and quality vectors:

[0096]

[0097]

[0098] Where g(⋅) is the weight generation network, This indicates vector concatenation.

[0099] (2) Window-level fusion representation: We get:

[0100]

[0101] (3) Subject-level background color aggregation: This involves combining the background colors of the same subject... The aggregation of window fusion representations yields:

[0102]

[0103] in This refers to the emotional undertone representation, used for multi-label prediction.

[0104] 4. Multi-label prediction and model training objectives

[0105] (1) Multi-label output: The multi-label prediction head with Sigmoid activation outputs the prediction vector.

[0106]

[0107] The confidence level corresponding to each background color label.

[0108] (2) Multi-label loss: using the binary cross-entropy (BCE) form:

[0109]

[0110] In engineering implementation, a method that combines Sigmoid and BCE and has a more stable value is BCEWithLogitsLoss (equivalent to calculating BCE after performing Sigmoid on logits).

[0111] (1) Stability regularization: To enhance the stability of the "background color" across windows, the following is introduced:

[0112]

[0113] (2) The total loss is:

[0114]

[0115] Among them, 𝜆1 and 𝜆2 are hyperparameters used to balance classification accuracy and stability.

[0116] Example 2:

[0117] This embodiment, based on the method described in Embodiment 1, provides an exemplary verification of the method of the present invention using the publicly available dataset AMIGOS. The AMIGOS dataset contains synchronously acquired facial videos and multi-source physiological signal data, including EEG, ECG, and GSR, and provides emotion annotation information such as valence, arousal, and control, which can be used to verify the effectiveness of the method of the present invention in multimodal emotion background multi-label recognition tasks.

[0118] In this embodiment, the AMIGOS dataset is divided into training, validation, and test sets based on the subjects, ensuring that subjects in the test set do not appear in the training set to reduce information leakage caused by subject overlap. Macro-F1 and mAP are used as evaluation metrics to comprehensively assess multi-label recognition performance. During training, the Adam optimizer is employed, with a learning rate set to... The batch size is set to 32, the number of training epochs is set to 50, and the Macro-F1 score on the validation set is used as the criterion for early stopping. The above training settings are the same as the experimental settings in your current draft.

[0119] To verify the effectiveness of the method of the present invention, it was compared with a single-modal method and a simple stitching and fusion method. The comparison methods included: a method using only the face video modality, a method using only the EEG modality, a method using both ECG and GSR modalities, a physiological multimodal method using both EEG, ECG, and GSR, and a simple stitching and fusion strategy.

[0120] Table 1 shows a set of exemplary results for each method on the AMIGOS dataset.

[0121] Table 1. A set of exemplary results on the AMIGOS dataset

[0122] As shown in Table 1, compared to single-modal methods, the method in this embodiment, which combines facial expressions and multi-source physiological signal information, achieves higher Macro-F1 and mAP, indicating that multimodal joint modeling helps improve the performance of multi-label recognition of emotional background. Compared to methods that simply concatenate or combine only physiological modalities, the method in this embodiment, through modality-weighted fusion based on quality vector constraints, can adaptively allocate fusion weights when there are fluctuations or inconsistent contribution levels among different modalities, thereby obtaining more stable window-level fusion representations and subject-level emotional background representations.

[0123] This embodiment demonstrates that by performing multimodal preprocessing, unified time axis mapping, and window-level synchronous sample extraction, followed by submodal feature extraction, quality vector construction, modal weighted fusion, and cross-window aggregation, the impact of single-modal noise, occlusion, artifacts, and cross-window fluctuations on the final recognition result can be reduced to a certain extent, thereby improving the robustness and interpretability of emotion background multi-label recognition.

[0124] Finally, although this specification describes embodiments, not every embodiment contains only one independent technical solution. Those skilled in the art should understand that the above embodiments can be combined, substituted, or equivalently transformed without departing from the concept of the present invention, and all such transformations should fall within the protection scope of the present invention.

Claims

1. A multi-label recognition method for emotion background based on the fusion of facial expressions and multi-source physiological signals, characterized in that, Includes the following steps: S1: Multimodal data acquisition, label reading, and window index construction: Acquire facial video data and multi-source physiological signal data corresponding to subject u, and read the emotion annotation information corresponding to the subject; construct a set of window indexes to be processed within the acquisition period according to the preset window length T and step size H, which is used to extract window-level synchronization samples from the preprocessed and time-synchronized data; the ground truth value of the emotion background multi-label is generated by the emotion annotation information according to the preset label construction rules; S2: Multimodal preprocessing, unified timeline mapping, and window-level synchronous sample extraction: Perform face detection, key point localization, face region alignment, and cropping on face videos to obtain preprocessed face sequences. ; Denoising filtering, artifact suppression, and amplitude normalization are performed on the multi-source physiological signals to obtain the preprocessed physiological signals. The video modalities are mapped to a unified timeline based on frame-level timestamps or preset frame rates. The physiological modalities are mapped to the unified timeline based on their respective sampling rates through resampling and / or interpolation. The corresponding data segments of each modality within the same time window are extracted using the window index described in S1 to obtain window-level synchronization samples. ; S3: Modal Feature Extraction and Quality Vector Construction: Construct a video modal encoder and a physiological submodal encoder respectively, and extract features from the window-level synchronization samples obtained in S2; the video modal encoder takes the face frame sequence within the window as input and outputs a video modal window-level representation; the physiological submodal encoder takes the window-level signal segments of each physiological submodal actually acquired as input and outputs a window-level representation of the corresponding physiological submodal; calculate at least one quality index for the video modality and each physiological submodal, and construct the corresponding modal quality vector q, which is used to generate the fusion weights of each modality; S4: Modal weighted fusion, cross-window aggregation and sentiment background multi-label prediction based on quality vector constraints: Based on the window-level representation of each modality and the corresponding quality vector, calculate the fusion weight of each modality in the current time window, and perform weighted fusion of the window-level representations of each modality to obtain the window-level fused representation; At the subject level, cross-window aggregation is performed on the fusion representations of multiple time windows to obtain the subject-level emotional background representation; then, the emotional background multi-label prediction results and their corresponding confidence scores are output through a multi-label prediction head, and the model parameters are jointly optimized using multi-label loss and stability constraints.

2. The multi-label recognition method for emotional background as described in claim 1, characterized in that, S1 includes: S1-1: The acquired multimodal data is defined as follows: The raw video modality data consists of facial video sequences of the subject during the acquisition period. The multi-source physiological signal data includes one or more of EEG, ECG, and GSR. The raw physiological modality data consists of EEG, ECG, and GSR signal sequences simultaneously acquired by the subject during the acquisition period. Emotional annotation information includes valence, arousal, control, and / or other emotion-related annotation information. in, A sequence of face frames aligned within a window. For each physiological signal segment within the window (m∈{EEG,ECG,GSR}); the truth value of the emotional background multi-label is defined as: Where L is the number of emotion background color labels; the true values ​​of the emotion background color multi-label are generated by the emotion annotation information according to the preset label construction rules; S1-2: The window index is constructed as follows: Let the unified time axis be t, the window length be T, and the step size be H. Then the time range of the k-th window is: Where T is the window length, H is the step size, and tu,0 is the start timestamp of subject u.

3. The multi-label recognition method for emotional background as described in claim 1, characterized in that, S2 includes: S2-1: Based on the window index described in S1, extract window-level synchronization samples from the preprocessed and time-aligned data. The data of each modal sample within the window are represented as follows: Physiological modality preprocessing: Filtering, artifact suppression, standardization, and missing fragment labeling were performed on EEG, ECG, and GSR, respectively; Video modal preprocessing: Detection, keypoint localization, alignment, cropping, and normalization are performed on video frames to obtain a sequence of face frames within a window. S2-2: Unified Timeline Mapping and Sample Extraction: Based on the video timestamp or frame rate and the sampling rate of each physiological modality, the data of each modality is mapped to a unified timeline, and window-level synchronous samples are extracted according to the window index described in S1.

4. The multi-label method for identifying emotional background as described in claim 1, characterized in that, In S3, the output of the modal encoder is defined as follows: a feature extraction network is established for each modality. This yields a window-level representation: 。 5. The multi-label recognition method for emotional background as described in claim 1, characterized in that, S4 includes: S4-1: Quality-perceived fusion can be represented as: First, calculate the fusion weights for each modality: Where g(·) is the weight generation network, and [·;·] denotes vector concatenation; then the window-level fusion representation is obtained: ； S4-2: Cross-window aggregation of emotional undertone representation is as follows: Let subject u have a total of Each window represents the underlying emotional tone. Defined as: S4-3: Multi-label prediction and confidence output is: Sentiment background color multi-label prediction vector: Where σ(·) is the Sigmoid function, and Wb is... For learnable parameters, The values ​​of each dimension can be used as the confidence level of the corresponding label; S4-4: The training objective function uses multi-label binary cross-entropy loss, and a stability regularization term can be added: To improve the cross-window stability of emotional background, a stability regularization term is introduced, defined as: And the total loss is obtained: 。

Images