A multi-modal multi-task emotion recognition method for faces obscured by head-mounted displays

Abstract

This invention discloses a multimodal, multi-task emotion recognition method for faces occluded by head-mounted displays. First, keyframes are identified, and regions of interest (ROIs) are located within these keyframes. Based on this, facial occlusion is applied to obtain training video sequences. Simultaneously, EEG signals from the ROIs and surrounding physiological signals are used as training signals to obtain training data. Then, a multimodal, multi-task emotion recognition model is constructed. A half-face encoding module with five 3D convolutional layers extracts spatial-temporal features from the video sequences. A physiological signal perception module is constructed using a dual-stream Transformer structure and an interactive modal fusion module to obtain physiological features, enhancing the emotion recognition operation applied to physiological signals. The half-face features are then concatenated with the physiological features to obtain the emotion feature F used for recognition. j Send to two classification layers CL V CL A This invention enhances emotion recognition during micro-expression events, achieving emotion prediction accuracy comparable to full-face images through the fusion of half-face and physiological signals, and possessing the ability to accurately identify complex emotional states.

Description

A multimodal, multitasking emotion recognition method for faces obscured by head-mounted displays Technical Field

[0001] This invention belongs to the field of emotion recognition technology, and more specifically, relates to a multimodal, multi-task emotion recognition method for faces obscured by head-mounted displays. Background Technology

[0002] Head-mounted displays (HMDs) are crucial for experiencing Extended Reality (XR), which includes Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). In this context, people's emotions are naturally and immediately evoked due to the high level of immersion and realism of XR. For example, virtual reality therapy studies the emotional responses of humans to treatments for mental health disorders such as phobias, obsessive-compulsive disorder, eating disorders, or social cognitive training. Furthermore, VR content is tailored to the user's emotional state to promote relaxation in the virtual world.

[0003] However, HMDs pose challenges to external recording technologies because they cover the upper half of the user's face. This limitation significantly impacts social XR applications, especially Facial Expression Recognition (FER), a key method for studying and distinguishing human emotions. Facial features play a crucial role in creating immersive user experiences in these applications. Discarding information from the upper half of the face reduces the accuracy of FER. Another difficulty in performing FER in immersive XR environments is that traditional FER methods rely on public databases containing complete facial information. To adapt these datasets to XR environments, some studies have made adjustments, such as placing a "VR patch" on the detected face and processing images of people wearing VR headsets into corresponding Grad-CAM interpretation masks.

[0004] Besides the problem of HMDs obscuring the face, the short duration of facial expressions—whether macro-expressions, which are voluntary muscle movements covering most of the face and lasting between 0.5 and 4 seconds, or micro-expressions, which are brief, involuntary facial changes, such as nasal wrinkles, lasting between 65 and 500 milliseconds, and their subtle movements—makes them difficult for humans to recognize. Human facial expressions can be a mixture of micro-expressions and macro-expressions. For example, surprise from both micro-expressions and macro-expressions involves raising eyebrows and widening eyes. Micro-expressions are virtually impossible to fake. Conversely, macro-expressions may fail to convey hidden emotions that determine true human feelings and mindset and can be easily controlled and manipulated.

[0005] Physiological responses, which are difficult to fake, provide a deeper understanding of underlying emotions. These responses originate from the central (brainstem) and autonomic nervous systems, which regulate bodily functions such as heart rate, respiration, blood pressure, swallowing, and pupillary movements. Electroencephalography (EEG) effectively reflects the electrical activity of the central nervous system and is correlated with emotion. EEG is frequently used in emotion recognition research in immersive virtual environments. Peripheral physiological signals, such as conductance skin response (GSR), photoplethysmography (PPG), or heart rate variability (HRV), can also be used to reliably measure emotional states and have been widely used in emotion engagement research in immersive virtual environments. Although physiological signals cannot be intentionally controlled or concealed, they are suitable for real-time emotion recognition. However, these signals can be very weak and easily contaminated by human factors such as noise. Therefore, identifying emotions using only physiological signals or incomplete facial expressions can be quite challenging. Some studies have used incomplete facial expressions and physiological signals, rather than just incomplete facial expressions, but this area still requires further research and exploration. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal, multi-task emotion recognition method for faces obscured by head-mounted displays, so as to solve the limitations of each individual modality and enhance the ability to accurately identify complex emotional states.

[0007] To achieve the above-mentioned objectives, this invention provides a multimodal, multi-task emotion recognition method for faces obscured by head-mounted displays, characterized by the following steps:

[0008] (1) Training data processing

[0009] 1.1) Select samples that simultaneously contain facial video and physiological signals as training data.

[0010] In the macro-expression dataset and the micro-expression dataset, multiple videos capturing the participants' faces, i.e., facial videos, along with corresponding electroencephalogram signals and surrounding physiological signals, were selected respectively.

[0011] 1.2) Facial video processing.

[0012] 1.2.1) Annotation-based segmentation

[0013] The selected facial video is divided into multiple n-second video segments: During the segmentation process, the first m seconds of facial video before stimulus presentation are extracted as the initial segment. Subsequently, starting from the m-second mark, i.e., when the stimulus begins, non-overlapping n-second segments are extracted. The number of segments depends on the duration of stimulus presentation. The final segment includes the last segment of the facial video, which is presented for less than n seconds. The initial segment and the last segment shorter than n seconds are discarded. This results in multiple n-second video segments. Each video segment is annotated, i.e., labeled with valence and arousal. Valence refers to the degree of pleasure, ranging from unpleasant to pleasant, while arousal is an indicator of the degree of physiological activation, ranging from calm to excited.

[0014] 1.2.2) Finding keyframes, i.e., Apex frames.

[0015] For each video segment, 68 facial markers are obtained in each frame. Based on the obtained facial markers, 10 regions on the face are defined, which represent areas where muscle movements occur frequently. The size of each region is half the width of the mouth.

[0016] Select the frame with the highest intensity difference, i.e., the largest change in intensity value, as the keyframe:

[0017] S i =f(Frame) i ,Frame onset )+f(Frame i ,Frame offset )

[0018]

[0019] in:

[0020]

[0021] Among them, S i This represents the change intensity value of the current frame, i.e., the i-th frame. The function f measures the absolute pixel difference between 10 regions between two frames, which is the sum of the absolute values ​​of the differences between corresponding pixel values, and then normalizes them. Frame i For the i-th frame of the video clip, Frame onset The starting frame of the video clip. offset The last frame of a video clip. i-∈For the preceding frames of the i-th frame, |Frame i -Frame * | represents the sum of the absolute differences between the pixel values ​​at corresponding positions in 10 regions between two frames, and * represents onset, offset, or i-∈. S represents i The largest frame number, i max Keyframe Index for identifying peak facial expression intensity apex ;

[0022] 1.2.3) Finding areas of interest

[0023] Select k frames around the keyframe as the region of interest. If the keyframe is at least k / 2 frames away from the edge of the video clip, then k / 2 frames are included on each side of it. However, if the keyframe is less than k / 2 frames away from any edge, then the frames on the side less than k / 2 frames are added from the opposite side. That is, if the edge is the last frame of the video clip, then the frames are added from the first frame. If the edge is the first frame of the video clip, then the frames are added from the last frame, until there are a total of k frames.

[0024] 1.2.4) Alignment and Cropping

[0025] For a video frame representing the region of interest, calculate the center coordinates x for the left and right eyes. left ,y left and x right ,y right And calculate the angle θ between the line connecting the centers of the left and right eyes and the horizontal direction:

[0026]

[0027] Then, using the center coordinates of the two eyes as the base point, rotate the video frame θ counterclockwise;

[0028] After the video frame is rotated, the coordinates of its pixels also need to be rotated accordingly:

[0029]

[0030] Where (xτ, yτ) are the coordinates of the rotated pixel, and (x, y) are the coordinates of the pixel before rotation. center ,y center () represents the center coordinates of the two eyes;

[0031] After aligning the face, the face is cropped to a fixed size based on the facial markers: the face is divided into three parts vertically: the middle part, which is the pixel distance from the center of the markers of the two eyes to the center of the markers of the mouth, accounts for 35% of the vertical dimension of the cropped face; the bottom part accounts for 35%; and the top part accounts for 30%. The horizontal part is cropped with the midpoint of the leftmost and rightmost markers as the center. The size of the cropped horizontal part is the same as the size of the vertical part.

[0032] Because the image has been cropped, the pixel coordinates need to be transformed again:

[0033]

[0034] Where (x″, y″) are the coordinates after cropping, and (x′, y′) are the coordinates before cropping. crop ,y′ crop () represents the coordinates of the top-left corner of the cropping rectangle. After converting the pixel coordinates, the video frame of the aligned region of interest is obtained.

[0035] 1.2.5) Artificial shading

[0036] For video frames of aligned regions of interest, facial occlusion is performed based on 68 markers detected in the keyframes: We initialize the occlusion patch size according to the head-mounted display, making its aspect ratio approximately 2:1, using the distance between the two temporal bones of the facial markers as the reference length, and then setting the midpoint between the center points of the left and right eyes as the center coordinates of the head-mounted display to generate the occlusion patch.

[0037] Based on the arctangent function of the changes in the y-coordinate and x-coordinate of the center points of the left and right eyes, the tilt angle is obtained. The center of the occlusion patch on the coordinate plane is taken as the pivot point, and the corner points of the occlusion patch are rotated accordingly to obtain the video frame of the face occlusion.

[0038] For each video segment, the process is performed according to steps 1.2.2) to 1.2.5) to obtain multiple video sequences with face occlusion and a length of k frames.

[0039] 1.3) Physiological signal processing

[0040] First, the EEG signal and surrounding physiological signal corresponding to the facial video are downsampled. Then, the EEG signal is filtered with a bandpass filter of 4.0-45.0Hz, the surrounding physiological signal corresponding to the facial video from the macro-expression dataset is filtered with a low-pass filter with a cutoff frequency of 60Hz, and the surrounding physiological signal corresponding to the facial video from the micro-expression dataset is not filtered.

[0041] The portion of the downsampled and filtered EEG signal and surrounding physiological signal in the region of interest of each video segment is extracted and used as the EEG signal and surrounding physiological signal of the respective video sequence, i.e., the region of interest of the video segment.

[0042] (2) Construct a multimodal, multi-task emotion recognition model

[0043] 2.1) A five-layer 3D convolutional layer is used to construct a half-face coding module, which is used to extract spatial-temporal features from a video sequence of length k frames with occluded faces, thus obtaining half-face features. Where j is the sequence number of the video segment;

[0044] 2.2) A physiological signal perception module is constructed using a dual-stream Transformer structure and an interactive modal fusion module. The dual-stream Transformer structure includes two convolutional blocks and two Transformer encoders. The two convolutional blocks are used to filter the EEG signals of the region of interest in the input video segment and the surrounding physiological signals, respectively. The filtered EEG signals and surrounding physiological signals are then fed into a Transformer encoder to obtain the EEG signal features of the video segment. and physiological signal characteristics around the video clip

[0045] The interactive modal fusion module includes two connectors, Ca1 and Ca2, and interactive extractors, IE1 and IE2. Connector Ca1 connects the two video clips to extract EEG signal features. Physiological signal characteristics around video clips Connect to obtain feature TE j Then feature TE j The data is sent to an interactive extractor IE1 to obtain feature IE1(TE). j Connector Ca2 will connect the two video clips to the EEG signal features. Physiological signal characteristics around video clips Connect to obtain feature TE j+1 Then feature TE j+1 The data is sent to an interactive extractor IE2 to obtain the feature IE2(TE). j+1 Then, the feature IE1(TE) j ) and feature ITE2 (TE j+1 The elements are summed one by one to obtain the fused feature IMF. j :

[0046]

[0047] in, This indicates that the elements are summed one by one;

[0048] Finally, the two fused features IMFs are combined. j Connect them to obtain physiological characteristics

[0049]

[0050] Where Ca represents the join operation;

[0051] 2.3) Half-face features and physiological characteristics Connecting them, we obtain the emotional feature F. j :

[0052]

[0053] Then emotional characteristic F j Send to two classifiers CL V CL A They were used to predict the valence and arousal of emotions, respectively.

[0054] (3) Training a multimodal, multi-task emotion recognition model

[0055] The cross-entropy loss function is used to measure the difference between the predictions and the true labels of the multimodal multi-task emotion recognition model. Specifically, the training data obtained in step (1) is fed into the multimodal multi-task emotion recognition model constructed in step (2), and loss functions L for valence and arousal are constructed. V and L A :

[0056]

[0057] Where N is the number of video segments in the training data. and These are the true labeled values ​​of valence and arousal for the j-th video segment, respectively. and These are the predicted probabilities of the true label values ​​of the predicted valence and arousal of the j-th video segment, respectively;

[0058] To minimize the total loss L = L V +L A Train the multimodal, multi-task emotion recognition model until the total loss L is less than a set threshold;

[0059] (4) Emotion recognition

[0060] The system collects facial videos of users, along with corresponding electroencephalogram (EEG) signals and surrounding physiological signals. These are then segmented into segments of n seconds each. The video segments are processed according to steps 1.2.2) to 1.2.4) to obtain video sequences. The EEG signals and surrounding physiological signals corresponding to the facial videos are downsampled. The EEG signals are then filtered using a bandpass filter of 4.0-45.0 Hz, and the surrounding physiological signals are filtered using a low-pass filter with a cutoff frequency of 60 Hz. The portions of the downsampled and filtered EEG signals and surrounding physiological signals within the region of interest (ROI) of each video segment are extracted and used as the EEG signals and surrounding physiological signals of their respective video sequences, i.e., the ROI of each video segment.

[0061] The EEG signals of the video sequence and the corresponding video segments of the region of interest, along with the surrounding physiological signals, are fed into the multimodal, multi-task emotion recognition model trained in step (3) to obtain the predicted probabilities of each valence and arousal. The valence and arousal corresponding to the highest predicted probability are the predicted valence and arousal.

[0062] The objective of this invention is achieved as follows.

[0063] This invention provides a multimodal, multi-task emotion recognition method for faces obscured by head-mounted displays. First, keyframes are identified, and regions of interest (ROIs) are located within these keyframes. Based on this, facial occlusion is applied to obtain training video sequences. Simultaneously, EEG signals from the ROIs and surrounding physiological signals are used as training signals to obtain training data. Then, a multimodal, multi-task emotion recognition model is constructed. A half-face encoding module, built with five 3D convolutional layers, extracts spatial-temporal features from the video sequences to obtain half-face features. A physiological signal sensing module is constructed using a dual-stream Transformer architecture and an interaction-based modal fusion module. This module aims to capture the heterogeneity and interactivity between EEG and physiological signals to enhance emotion recognition operations applied to physiological signals. Physiological features are obtained through the physiological signal sensing module. Half-face features Physiological characteristics The connection is used to obtain the emotion feature F for recognition. j Send to two classification layers CL V CL A This invention enhances emotion recognition during micro-expression events, achieving emotion prediction accuracy comparable to full-face images through the fusion of half-face and physiological signals, and possessing the ability to accurately identify complex emotional states. Attached Figure Description

[0064] Figure 1 is a flowchart of the multimodal, multi-task emotion recognition method for faces obscured by head-mounted displays according to the present invention;

[0065] Figure 2 is a network architecture diagram of the multimodal, multi-task emotion recognition method for faces obscured by head-mounted displays according to the present invention.

[0066] Figure 3 is a specific example of facial video processing, where (a) shows the acquisition of 68 facial key points and 10 key regions, (b) shows alignment, (c) shows alignment, and (d) shows the addition of head-mounted display (HMD) occlusion. Detailed Implementation

[0067] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0068] Figures 1 and 2 are flowcharts and network architecture diagrams of the multimodal, multi-task emotion recognition method for faces obscured by head-mounted displays according to the present invention.

[0069] In this embodiment, as shown in Figures 1 and 2, the multimodal multitasking emotion recognition method for faces obscured by a head-mounted display of the present invention includes the following steps:

[0070] Step S1: Training Data Processing

[0071] Step S1.1: Select samples that simultaneously contain facial video and physiological signals as training data.

[0072] In the macro-expression dataset and micro-expression dataset, multiple videos capturing the participants' faces, i.e., facial videos, along with corresponding electroencephalogram (EEG) signals and surrounding physiological signals, were selected.

[0073] In this embodiment, the macro-expression dataset was selected from the AMIGOS dataset, in which 37 out of 40 participants (excluding participants 8, 24, and 28) participated in short and long video experiments designed to elicit emotional responses. Their facial videos, EEG, and surrounding physiological signals were fully recorded by three annotators.

[0074] The microexpression dataset was selected from the DEAP dataset, in which complete facial videos, EEG, and surrounding physiological signals were recorded by one annotator for the first 22 participants (out of 32). However, due to technical issues, some of the final trial data for participants 3, 5, 11, and 14 were lost.

[0075] Step S1.2: Facial video processing

[0076] Step S1.2.1: Annotation-based segmentation

[0077] The selected facial video was divided into multiple n-second video segments: During the segmentation process, the first m seconds of facial video before the stimulus presentation were extracted as the initial segment. Subsequently, starting from the m-second mark, i.e., when the stimulus began, non-overlapping n-second segments were extracted. The number of segments depended on the duration of the stimulus presentation. The final segment included the last segment of the facial video, which was presented for less than n seconds. The initial segment and the last segment shorter than n seconds were discarded. This resulted in multiple n-second video segments. Each video segment was annotated, i.e., labeled with valence and arousal. Valence refers to the degree of pleasure, ranging from unpleasant to pleasant, while arousal is an indicator of the degree of physiological activation, ranging from calm to excited.

[0078] In the AMIGOS dataset, videos capturing participants' faces were divided into multiple segments of n = 20 seconds each. The first m = 5 seconds before stimulus presentation were extracted as the initial segments. Subsequently, non-overlapping 20-second segments were extracted starting from the m = 5-second mark, i.e., the start of the stimulus, with the number of segments depending on the duration of the stimulus video. The final segment included the last part of the video, which was shorter than n = 20 seconds. To ensure consistency in segment length, the first and last segments shorter than n = 20 seconds were discarded. Three annotators provided external annotations of emotion and arousal for the 20-second segments of facial videos in both short and long video experiments. The valence and arousal assigned by the three annotators were averaged, and the average valence and arousal were used as the label for the segment. In the DEAP dataset, each facial video lasted approximately 60 seconds; since there was only one annotator for this duration, no averaging was performed.

[0079] Step S1.2.2: Find the keyframe, i.e., the Apex frame.

[0080] For each video segment, 68 facial markers are acquired in each frame. Based on the acquired facial markers, 10 regions on the face are defined, which represent areas where muscle movements occur frequently. The size of each region is half the width of the mouth.

[0081] Select the frame with the highest intensity difference, i.e., the largest change in intensity value, as the keyframe:

[0082] S i =f(Frame) i ,Frame onset )+f(Frame i ,Frame offset )

[0083]

[0084] in:

[0085]

[0086] Among them, S i This represents the change intensity value of the current frame, i.e., the i-th frame. The function f measures the absolute pixel difference between 10 regions between two frames, which is the sum of the absolute values ​​of the differences between corresponding pixel values, and then normalizes them. Frame i For the i-th frame of the video clip, Frame onse The starting frame of the video clip. offset The last frame of a video clip. i-∈ For the preceding frames of the i-th frame, |Frame i -Frame * | represents the sum of the absolute differences between the pixel values ​​at corresponding positions in 10 regions between two frames, and * represents onset, offset, or i-∈. S represents i The largest frame number, i max Keyframe Index for identifying peak facial expression intensity apex .

[0087] In this embodiment, an open-source facial toolkit was used to acquire 68 facial markers in each frame of the video clip, as shown in Figure 3. However, due to factors such as poor video quality and low lighting, some videos did not detect faces, with successful detection occurring in approximately 70% of the videos. Based on previous research, we defined 10 regions on the face, as shown in the rectangles in Figure 3, representing areas where muscle movement frequently occurs, based on the detected markers. The size of each region was estimated to be half the width of the mouth. These regions are depicted in Figure 1. The facial video sequence begins at the onset frame and ends at the offset frame. The keyframe, or Apex frame, is the frame in the sequence with the most significant intensity change. To identify keyframes, this invention calculates the absolute pixel differences between the current frame and the onset and offset frames in the ten regions. Simultaneously, to minimize environmental noise, normalization is performed by dividing the sum of the differences by the differences between the considered frame and its consecutive frames. A keyframe should represent the peak of the intensity difference with the onset and offset frames. Therefore, this invention selects the frame with the highest per-pixel value of the intensity difference as the keyframe. In this embodiment, the deviation is ∈ = 3. Keyframes are crucial for facial expression recognition tasks.

[0088] Step S1.2.3: Find the region of interest

[0089] Select k frames around the keyframe as the region of interest. If the keyframe is at least k / 2 frames away from the edge of the video clip, then k / 2 frames are included on each side of it. However, if the keyframe is less than k / 2 frames away from any edge, then frames on the side less than k / 2 frames are added from the opposite side. That is, if the edge is the last frame of the video clip, then frames are added from the first frame; if the edge is the first frame of the video clip, then frames are added from the last frame, and so on, until there are a total of k frames.

[0090] Given that the frame rates of the two datasets are 25fps and 50fps respectively, we reduce any video with a frame rate higher than 25fps to 25fps to maintain consistency, taking k=100, i.e., collecting 100 frames over 4 seconds. We then select the 100 frames (4 seconds) surrounding the keyframe as the region of interest (ROI). If the keyframe is at least 2 seconds away from the edge of the video clip, we include 2 seconds (50 frames) on each side of it. However, if the keyframe is less than 2 seconds away from any edge, we add frames from the opposite side until a total of 100 frames are reached.

[0091] Step S1.2.4: Alignment and Cropping

[0092] After detecting facial key points, we align the face based on the eye key points by rotating the image, as shown in Figure 3.

[0093] For a video frame representing the region of interest, calculate the center coordinates x for the left and right eyes. left ,y left and x right ,y right And calculate the angle θ between the line connecting the centers of the left and right eyes and the horizontal direction:

[0094]

[0095] Then, using the center coordinates of the two eyes as the base point, rotate the video frame θ counterclockwise.

[0096] After the video frame is rotated, the coordinates of its pixels also need to be rotated accordingly:

[0097]

[0098] Where (x′, y′) are the coordinates of the rotated pixel, and (x, y′) are the coordinates of the pixel before rotation. center ,y center () are the center coordinates of the two eyes.

[0099] After aligning the face, the face is cropped to a fixed size based on the facial markers: the face is divided into three parts vertically: the middle part, which is the pixel distance from the center of the markers of the two eyes to the center of the marker of the mouth, accounts for 35% of the vertical dimension of the cropped face; the bottom part accounts for 35%; and the top part accounts for 30%. The horizontal part is cropped with the midpoint of the leftmost and rightmost markers as the center. The size of the cropped horizontal part is the same as the size of the vertical part.

[0100] Because the image has been cropped, the pixel coordinates need to be transformed again:

[0101]

[0102] Where (x″, y″) are the coordinates after cropping, and (x′, y′) are the coordinates before cropping. crop ,y′ crop () represents the coordinates of the top-left corner of the cropping rectangle. After converting the pixel coordinates, the video frames of the aligned region of interest are obtained.

[0103] Step S1.2.5: Manual occlusion

[0104] For video frames of aligned regions of interest, facial occlusion is performed based on 68 markers detected in the keyframes: We initialize the occlusion patch size according to the head-mounted display, making its aspect ratio approximately 2:1, using the distance between the two temporal bones of the facial markers as the reference length, and then setting the midpoint between the center points of the left and right eyes as the center coordinates of the head-mounted display to generate the occlusion patch.

[0105] The tilt angle is obtained by using the arctangent function of the changes in the y-coordinate and x-coordinate of the center points of the left and right eyes. The center of the occlusion patch on the coordinate plane is taken as the pivot point, and the corner points of the occlusion patch are rotated accordingly to obtain the video frame of the face occlusion.

[0106] In this embodiment, a rotation matrix is ​​used to rotate the corner points of the occlusion patch according to its central pivot point in the coordinate plane. This geometric model provides a more realistic occlusion effect because it is based on wearing a head-mounted display, rather than simply covering the upper half of the face.

[0107] For each video segment, the process is performed according to steps S1.2.2 to S1.2.5 to obtain multiple video sequences with face occlusion and a length of k frames.

[0108] S1.2: Physiological Signal Processing

[0109] First, the EEG signals and surrounding physiological signals corresponding to the facial videos are downsampled. Then, the EEG signals are filtered using a bandpass filter of 4.0-45.0 Hz, the surrounding physiological signals corresponding to the facial videos from the macro-expression dataset are filtered using a low-pass filter with a cutoff frequency of 60 Hz, and the surrounding physiological signals corresponding to the facial videos from the micro-expression dataset are not filtered.

[0110] The downsampled and filtered EEG signals and surrounding physiological signals are extracted from the regions of interest (ROIs) of each video segment and used as the EEG signals and surrounding physiological signals of their respective video sequences, i.e., the ROIs of the video segments.

[0111] In this embodiment, the peripheral physiological signal selected is the skin conductance response (GSR) signal.

[0112] Step S2: Construct a multimodal, multi-task emotion recognition model

[0113] Step S2.1: Construct the half-face coding module

[0114] As shown in Figure 2, a five-layer 3D convolutional layer is used to construct a half-face coding module, which is used to extract spatial-temporal features from a video sequence of length k frames with face occlusion, thus obtaining half-face features. Where j is the sequence number of the video segment. In this embodiment, half-face features...

[0115] Step S2.2: Construct a physiological signal sensing module

[0116] As shown in Figure 2, a physiological signal perception module is constructed using a dual-stream Transformer structure and an interactive modal fusion module. The dual-stream Transformer structure includes two convolutional blocks and two Transformer encoders. The two convolutional blocks are used to filter the EEG signal and surrounding physiological signals of the region of interest in the input video segment, respectively. The filtered EEG signal and surrounding physiological signals are then fed into separate Transformer encoders to obtain the EEG signal features of the video segment. and physiological signal characteristics around the video clip Since the surrounding physiological signals selected are skin conductance response (GSR) signals, the characteristics of the surrounding physiological signals in the video clips are... Recorded as Transformer encoders (TEs). These TEs employ a multi-head attention mechanism to capture the dependencies between elements of the input feature map. This mechanism is defined as follows:

[0117] Multi-Head(Q,K,V)=Concat(head1,…,head h )

[0118] Where Q, K, and V are the query, key, and value matrix, and h is the number of heads, each head being calculated as:

[0119] head i =Attention(QW i Q ,KW i K VW i V )

[0120] Among them, W i This represents the weight matrix for Q, K, and V. The outputs are then concatenated and projected back to the original dimensions. The Transformer encoder is existing technology and will not be described further here.

[0121] The interactive modal fusion module includes two connectors, Ca1 and Ca2, and interactive extractors, IE1 and IE2. Connector Ca1 connects the two video clips to extract EEG signal features. Physiological signal characteristics around video clips Connect to obtain feature TE j Then feature TE j The data is sent to an interactive extractor IE1 to obtain feature IE1(TE). j Connector Ca2 will connect the two video clips to the EEG signal features. Physiological signal characteristics around video clips Connect to obtain feature TE j+1 Then feature TE j+1 The data is sent to an interactive extractor IE2 to obtain the feature IE2(TE). j+1 Then, the feature IE1(TE) j ) and feature IE2 (TE) j+1 The elements are summed one by one to obtain the fused feature IMF. j :

[0122]

[0123] in, This indicates that the elements are summed one by one.

[0124] Finally, the two fused features IMFs are combined. j Connect them to obtain physiological characteristics

[0125]

[0126] Here, Ca represents the join operation.

[0127] In this embodiment, the obtained physiological characteristics

[0128] Step S2.3: Extract half-face features and physiological characteristics Connecting them, we obtain the emotional feature F. j :

[0129]

[0130] Then emotional characteristic F j Send to two classifiers CL V CL A They were used to predict the valence and arousal of emotions, respectively.

[0131] Step S3: Train a multimodal, multi-task emotion recognition model

[0132] The cross-entropy loss function is used to measure the difference between the predictions and the true labels of the multimodal, multi-task emotion recognition model. Specifically, the training data obtained in step S3 is fed into the multimodal, multi-task emotion recognition model constructed in step S2, and loss functions L for valence and arousal are constructed. V and L A :

[0133]

[0134] Where N is the number of video segments in the training data. and These are the true labeled values ​​of valence and arousal for the j-th video segment, respectively. and These are the predicted probabilities of the true label values ​​of the predicted valence and arousal of the j-th video segment, respectively.

[0135] To minimize the total loss L = L V +L A Train the multimodal, multi-task emotion recognition model until the total loss L is less than a set threshold.

[0136] Step S4: Emotion Recognition

[0137] The system acquires facial video of the user, along with corresponding EEG signals and surrounding physiological signals. The video segments are then divided into segments of n seconds each. The video segments are processed according to steps S1.2.2–S1.2.4 to obtain video sequences. The EEG signals and surrounding physiological signals corresponding to the facial video are downsampled. The EEG signals are then filtered using a 4.0–45.0 Hz bandpass filter, and the surrounding physiological signals are filtered using a 60 Hz low-pass filter. The portions of the downsampled and filtered EEG signals and surrounding physiological signals within the region of interest (ROI) of each video segment are extracted and used as the EEG signals and surrounding physiological signals for their respective video sequences, i.e., the ROI of each video segment.

[0138] The video sequence and the corresponding video segment region of interest's EEG signal and surrounding physiological signals are fed into the multimodal multi-task emotion recognition model trained in step S3 to obtain the predicted probabilities of each valence and arousal. The valence and arousal corresponding to the highest predicted probability are the predicted valence and arousal.

[0139] Experiments on the validation set

[0140] 1. Comparison of keyframes and adjacent frames with keyframes only

[0141] The accuracy and F1 score of this invention outperform the ResNet50 method for facial expressions under partial occlusion on the AMIGOS dataset, which uses a single frame as input and improves by 5.69% and 6.35%, respectively, and by 5.47% and 10.98% on the DEAP dataset, as shown in Table 1.

[0142]

[0143] Table 1

[0144] Table 1 compares the facial expression recognition performance on the AMIGOS (AM) and DEAP (D) datasets. Note: Full face (FF), half face (HF), physiological signal (P), emotional valence (V), emotional arousal (A), and facial expression recognition performance under partial occlusion (FERPO).

[0145] Furthermore, the inference time is suitable for real-time tasks. This is thanks to the superiority of our method in handling occluded facial information and fusing physiological signals. Continuous signal frames capture more temporal information, mitigating the impact of single-frame noise or loss of detail. This invention not only enhances the performance of partially occluded facial expression recognition (FERPO) in XR HMDs but also reduces prediction time, making it ideal for real-time XR applications. Moreover, the multi-task method of this invention offers more advantages than the single-task method, including fewer parameters, faster convergence, and improved performance. Additionally, micro-expression datasets performed worse than macro-expressions in tests, possibly due to their more subtle facial changes. Therefore, for HMD users, fusing physiological signals with facial data can enhance emotion recognition when micro-expressions occur.

[0146] 2. Comparison of full-face and half-face physiological data

[0147] Simultaneous emotion and arousal prediction using half-face images and physiological data achieves higher accuracy, improved F1 score, and faster results compared to full-face images. This means that the fusion of half-face and physiological signals achieves emotion prediction accuracy comparable to that of full-face images.

[0148] 3. Ablation Research

[0149] The validation objective was to conduct a detailed analysis of the contributions of visual and physiological modalities in the proposed method. Table 1 shows that FERPOphysio-only performs significantly worse than FERPOhalf-face-only and FERPOfull on the AMIGOS and DEAP datasets. This is because FERPOphysio-only uses only physiological data, ignoring facial information that provides details about the subject's emotional expression. As a result, it leads to inaccurate results for emotion and arousal when the limited physiological signals may not accurately determine the emotional state. This highlights the importance of facial information in improving the accuracy of emotional state prediction.

[0150] Furthermore, FERPOhalf-face-only performed significantly worse than FERPOfull. This is because FERPOhalf-face-only only uses details from half of the face, ignoring the spatial relationships of key points on the complete face. Therefore, this greatly increases the difficulty of predicting positive emotions, negative emotions, and arousal levels, leading to less accurate emotion and arousal results. This underscores the importance of supplementing with additional physiological modalities to address the information incompleteness caused by partial occlusion. Including physiological information significantly enhances the model's ability to accurately identify complex emotional states.

[0151] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A multimodal, multi-task emotion recognition method for faces obscured by head-mounted displays, characterized in that, Includes the following steps: (1) Training data processing; 1.1) Select samples that simultaneously have facial video and physiological signals as training data; In the macro-expression dataset and micro-expression dataset, multiple videos capturing participants' faces, i.e., facial videos, along with corresponding EEG signals and surrounding physiological signals, were selected; 1.2) Facial video processing; 1.2.1) Annotation-based segmentation; The selected facial videos were divided into multiple... A 10-second video clip: During the segmentation process, the period before stimulus presentation... A second of facial video was extracted as the initial segment, and then... At the start of the second marker, i.e., when the stimulus begins, non-overlapping signals are extracted. The number of segments depends on the duration of stimulus presentation. The final segment includes the last segment of the facial video, which has fewer than [number] stimulus presentations. seconds, discard the initial fragment and the last one shorter than A segment of seconds, thus obtaining multiple Each video clip, measured in seconds, is annotated with labels for valence and arousal. Valence refers to the degree of pleasure, ranging from unpleasant to pleasant, while arousal is an indicator of physiological activation, ranging from calm to excited. 1.2.2) Keyframes, or Apex frames, are identified. For each video clip, 68 facial markers are acquired from each frame. Based on these markers, 10 regions on the face are defined, representing areas where muscle movement frequently occurs. Each region is half the width of the mouth. The frame with the highest intensity difference (i.e., the largest change in intensity value) is selected as the keyframe. ;in: ； ;in, Indicates the current frame, i.e., the first frame. The change intensity value of the frame, function To measure the absolute pixel difference of pixel values ​​in 10 regions between two frames, which is the sum of the absolute values ​​of the differences in pixel values ​​at corresponding positions, and then normalize the result, The first video clip frame, This is the starting frame of the video clip. This is the end frame of the video clip. For the i-th frame frame, This represents the sum of the absolute differences between the pixel values ​​at corresponding positions in 10 regions between two frames. * indicates... , or ， express The largest frame number, i.e. Keyframes for identifying peak facial expression intensity ;1.2.3) Find the region of interest; select the region surrounding the keyframe A frame is considered a region of interest if the keyframe is at least [distance] from the edge of the video clip. / 2 frames, then each side contains / 2 frames, but if the keyframe is less than 2 frames away from any edge. If the edge is the last frame of a video clip, then frames smaller than one side are added from the opposite side. That is, if the edge is the last frame of the video clip, then frames are added from the first frame; if the edge is the first frame of the video clip, then frames are added from the last frame, and so on, until a total of [number missing] frames are added. Frame; 1.2.4), Alignment and Cropping; For video frames with a region of interest, calculate the center coordinates of the left and right eyes. , and , And calculate the angle between the line connecting the centers of the left and right eyes and the horizontal direction. ： Then, using the center coordinates of the two eyes as the base point, rotate the video frame counterclockwise. After the video frame is rotated, the coordinates of its pixels also need to be rotated accordingly. ;in, These are the coordinates of the rotated pixel. These are the pixel coordinates before rotation. These are the center coordinates of the two eyes; after aligning the face, the face is cropped to a fixed size based on the facial markers: vertically, the face is divided into three parts: the middle part, which is the pixel distance from the center of the markers for both eyes to the center of the marker for the mouth, accounting for 35% of the vertical dimension of the cropped face; the bottom part accounts for 35%; and the top part accounts for 30%. Horizontally, the cropping is performed with the midpoint between the leftmost and rightmost markers as the center, and the horizontal size after cropping is the same as the vertical size. Because the image is cropped, the pixel coordinates need to be transformed again. ;in,( ) are the coordinates after cropping, ( ) are the coordinates before cropping, ( ) is the coordinate of the top left corner of the cropping rectangle. After converting the pixel coordinates, the video frame with the aligned region of interest is obtained; 1.2.5), artificial occlusion; For the video frame with the aligned region of interest, facial occlusion is performed based on 68 marker points detected in the keyframe: We initialize the occlusion patch size according to the head-mounted display, making its aspect ratio 2:1, using the distance between the two temporal bones of the facial marker points as the reference length, and then setting the midpoint between the center points of the left and right eyes as the center coordinates of the head-mounted display to generate the occlusion patch; based on the center points of the left and right eyes Coordinate transformation and The arctangent function of the coordinate transformation is used to obtain the tilt angle. The center of the occlusion patch on the coordinate plane is used as the pivot point, and the corner points of the occlusion patch are rotated accordingly to obtain the video frame of the face occlusion. Each video segment is processed according to steps 1.2.2) to 1.2.5) to obtain multiple face occlusion frames of length [missing information]. Frame video sequence; 1.3) Physiological signal processing; First, the EEG signal and surrounding physiological signal corresponding to the facial video are downsampled. Then, the EEG signal is filtered with a bandpass filter of 4.0-45.0Hz. The surrounding physiological signal corresponding to the facial video from the macro-expression dataset is filtered with a low-pass filter of 60Hz cutoff frequency. The surrounding physiological signal corresponding to the facial video from the micro-expression dataset is not filtered. The parts of the downsampled and filtered EEG signal and surrounding physiological signal in the region of interest of each video segment are extracted as the EEG signal and surrounding physiological signal of the region of interest of each video sequence, i.e., the video segment region of interest; (2) Construct a multimodal multi-task emotion recognition model; 2.1) A half-face coding module is constructed using a five-layer 3D convolutional layer to extract spatial-temporal features from a video sequence of length k frames with facial occlusion, and obtain half-face features. ,in, 2.2) A physiological signal perception module is constructed using a dual-stream Transformer structure and an interactive modal fusion module. The dual-stream Transformer structure includes two convolutional blocks and two Transformer encoders. The two convolutional blocks are used to filter the EEG signal and surrounding physiological signals of the region of interest in the input video segment, respectively. The filtered EEG signal and surrounding physiological signals are then sent to a Transformer encoder to obtain the EEG signal features of the video segment. and physiological signal characteristics around the video clip The interactive modal fusion module includes two connectors, Ca1 and Ca2, and interactive extractors, IE1 and IE2. Connector Ca1 will extract the EEG signal features from the video clips. Physiological signal characteristics around video clips Connect to obtain features Then the features The data is sent to an interactive extractor IE1 to obtain features. Connector Ca2 will combine the two video clips with EEG signal features. Physiological signal characteristics around video clips Connect to obtain features Then the features The data is sent to an interactive extractor (IE2) to obtain features. Then, the features With features By summing each element individually, the fusion feature is obtained. ： ;in, This involves summing each element individually; finally, the two features are merged. Connect them to obtain physiological characteristics ： ; where Ca represents the join operation; 2.3), half-face features and physiological characteristics Connect them to obtain emotional characteristics ： Then, emotional characteristics Send to two classifiers CL V CL A (2) Predict the valence and arousal of emotions respectively; (3) Train a multimodal multi-task emotion recognition model; Use the cross-entropy loss function to measure the difference between the prediction and the real label of the multimodal multi-task emotion recognition model. Specifically, send the training data obtained in step (1) into the multimodal multi-task emotion recognition model constructed in step (2), and construct the loss functions for valence and arousal. and ： ;in, It is the number of video clips in the training data. and They are the first The true labeled values ​​of valence and arousal of a video clip. and They are the first The predicted probabilities of the true labeled values ​​of valence and arousal for each video segment are used to minimize the total loss. Train the multimodal, multi-task emotion recognition model until the total loss is reached. Less than the set threshold; (4) Emotion recognition; Collect the user's facial video and the corresponding EEG signal and surrounding physiological signal, and divide it into segments of n seconds each. Then process the video segments according to steps 1.2.2) to 1.2.4) to obtain the video sequence. Downsample the EEG signal and surrounding physiological signal corresponding to the facial video. Then filter the EEG signal with a bandpass filter of 4.0-45.0Hz and filter the surrounding physiological signal with a low-pass filter with a cutoff frequency of 60Hz. Take out the part of the downsampled and filtered EEG signal and surrounding physiological signal in the region of interest of each video segment as the EEG signal and surrounding physiological signal of the region of interest of each video segment. Send the video sequence and the EEG signal and surrounding physiological signal of the region of interest of the corresponding video segment into the multimodal multi-task emotion recognition model trained in step (3) to obtain the predicted probability of each valence and arousal. The valence and arousal corresponding to the highest predicted probability are the predicted valence and arousal.

2. The multimodal, multi-task emotion recognition method for faces obscured by a head-mounted display according to claim 1, characterized in that, =20， =3， =100。

Images