An occluded face expression recognition method and system based on a dynamic double-flow network

Abstract

The present application relates to the technical fields of computer vision and artificial intelligence, in particular to a kind of occluded face expression recognition method and system based on dynamic double-flow network.The method comprises: obtaining a face image to be identified;Face image is simultaneously input into parallel face expression stream and occlusion perception stream to extract global expression feature vector and occlusion robust expression feature vector;Global expression feature vector and occlusion robust expression feature vector are input into dynamic weight generator to generate dynamic fusion weight;Global expression feature vector and occlusion robust expression feature vector are weighted and fused to obtain final feature vector;Final feature vector is input into classifier to obtain expression classification result.The present application utilizes occlusion information as an effective feature, rather than simply ignoring or repairing, significantly improves the recognition accuracy and robustness under the existence of mask, glasses and other occlusions, and can be widely applied in intelligent security and human-computer interaction fields.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and more specifically to a method and system for recognizing occluded facial expressions based on a dynamic dual-stream network. Background Technology

[0002] Facial Expression Recognition (FER) technology is a core component of affective computing and has broad application prospects in fields such as human-computer interaction, intelligent security, mental health monitoring, autonomous driving, and online education. Although deep learning-based methods have achieved near-human level recognition accuracy in controlled laboratory environments, their performance still drops sharply in complex real-world scenarios. A major challenge causing this problem is the ubiquitous and unpredictable issue of facial occlusion, such as masks, glasses, sunglasses, hands, hair, hats, and accessories.

[0003] Currently, the mainstream solutions for facial expression recognition under occlusion can be roughly divided into three categories:

[0004] (1) Local feature enhancement method: This method forces the model to focus on the unoccluded facial regions (such as eyes and mouth) by designing network structures (such as introducing local connection layers) or loss functions. However, the position, size and shape of the occluders are random, and the model has difficulty in pre-defining or adaptively locating all possible occlusion situations, resulting in limited generalization ability.

[0005] (2) Occlusion Removal / Repair Method: This method first uses Generative Adversarial Network (GAN) or image inpainting technology to fill in or repair the occluded areas in the input image, and then sends the "complete" face image into the recognition model. The main drawback of this method is that the computational cost is huge, and the repair process may introduce distorted textures and features. These noises will interfere with subsequent recognition tasks, leading to performance degradation.

[0006] (3) Attention mechanism method: This method introduces attention modules (such as SE module, self-attention mechanism) to allow the model to automatically learn important facial feature regions. However, when faced with large-area or strong interference occlusion, the attention map is prone to become chaotic or unstable and cannot reliably focus on effective facial feature regions.

[0007] In summary, most existing technologies attempt to ignore, bypass, or repair occlusions, treating them as mere interference signals. This methodological limitation makes it difficult for models to effectively handle complex and ever-changing real-world occlusion scenarios. Therefore, this invention proposes an occluded facial expression recognition method based on a dynamic dual-stream network to address the aforementioned problems. Summary of the Invention

[0008] To overcome the aforementioned deficiencies of the prior art, the present invention provides a method and system for occluded facial expression recognition based on a dynamic dual-stream network, in order to solve the problems existing in the background art.

[0009] According to a first aspect of the present invention, an occluded facial expression recognition method based on a dynamic dual-stream network is provided. The method includes: acquiring a facial image to be recognized; simultaneously inputting the facial image into a parallel facial expression stream and an occlusion perception stream; the facial expression stream being used to extract a global expression feature vector. The occlusion-aware stream is used to extract occlusion-robust facial feature vectors by segmenting the occluded region and guiding attention to the unoccluded facial region based on an attention mechanism. ; the global facial expression feature vector and the occlusion robust facial feature vector Input to the dynamic weight generator to generate dynamic fusion weights. ,in Using formulas For the global facial expression feature vector and the occlusion robust facial feature vector Weighted fusion is performed to obtain the final feature vector. The final feature vector The input is fed into the classifier to obtain the expression classification results.

[0010] This embodiment significantly improves the accuracy and overall robustness of facial expression recognition under occlusion. By processing global information and local occlusion robustness information in parallel, the model obtains more comprehensive judgment criteria and no longer fails due to the lack of local information. The solution also achieves adaptive processing capability for occlusion scenarios. The model can intelligently cope with various random occlusions without pre-setting the occlusion mode.

[0011] In some embodiments, the occlusion-aware stream extracts the occlusion-robust facial feature vector. The method includes: segmenting the input face image using an occlusion region and an importance-aware network to generate a multi-channel mask image; wherein, the first channel of the multi-channel image is a binary mask used to identify the occlusion region. At least one additional channel in the multi-channel array is used to represent the importance weight of facial regions for expression recognition. According to the importance weight mask With the binary mask Generate soft attention maps for unoccluded regions ,in This represents element-wise multiplication; utilizing the soft attention map. The multi-level intermediate feature maps in the occlusion-aware stream are modulated; the occlusion-robust facial feature vector is extracted from the modulated feature maps using a feature extraction network. .

[0012] In this embodiment, firstly, by introducing region importance prediction, the model's attention mechanism is upgraded from passively avoiding occlusion to actively seeking important and unoccluded regions, achieving more intelligent and refined feature focusing and enhancing the discriminative ability of the extracted features. Secondly, the generated soft attention map... Compared to simple binary mask hard attention, it can distribute feature weights more smoothly and reasonably, avoiding boundary effects and information loss, and further improving feature quality and model performance.

[0013] In some embodiments, the occlusion region and importance-aware network includes a fully convolutional network based on an encoder-decoder structure.

[0014] This embodiment ensures the accuracy of occluded region segmentation. The encoder-decoder structure effectively fuses low-level detail information and high-level semantic information to generate pixel-accurate masks, providing a reliable guarantee for subsequent feature modulation and avoiding feature misleading caused by segmentation errors.

[0015] In some embodiments, the global facial feature vector and the occlusion robust facial feature vector Input to the dynamic weight generator to generate dynamic fusion weights. This includes: converting the global facial expression feature vector The occlusion robust facial feature vector and the occlusion statistical feature vector of the occlusion region extracted from the occlusion sensing stream. The dynamic weight generator is input, and the dynamic fusion weights are output after being calculated by a lightweight neural network module. .

[0016] In some embodiments, the occlusion statistical feature vector The binary mask output by the occluded region segmentation network is determined using the following method. The data is analyzed, statistical features are extracted and encoded into a fixed-length vector as the occlusion statistical feature vector. The statistical features include at least one of the following: the percentage of the total area of ​​the occluded region, the coverage ratio of the occluded region in the eye, the coverage ratio of the occluded region in the mouth, and the coverage ratio of the occluded region in the eyebrow.

[0017] This embodiment is based on quantifiable physical statistics such as occlusion area and location. This allows the model to intelligently adjust its trust in the global flow and occlusion-aware flow based on the severity and type of occlusion. The decision-making process is transparent, credible, and intuitive. At the same time, introducing these explicit features also helps the neural network make more accurate and stable decisions.

[0018] In some embodiments, the lightweight neural network module includes a multilayer perceptron model, wherein the generation of dynamic fusion weights... This includes: converting the global facial expression feature vector The occlusion robust facial feature vector and the occlusion statistical feature vector The concatenated vector is input into the multilayer perceptron model, and the dynamic fusion weights are output through the sigmoid activation function of the multilayer perceptron model. The dynamic fusion weight It is a scalar.

[0019] In some embodiments, the facial expression stream and the occlusion perception stream share convolutional weights in a shallow network.

[0020] This embodiment reduces the number of model parameters and computational cost, improving training and inference efficiency. The two streams do not need to independently learn completely identical low-level features (such as edges and textures), and sharing shallow weights avoids parameter redundancy, making the model more lightweight. At the same time, this scheme also enhances the model's generalization ability. Sharing weights forces the network to learn more general low-level feature representations that are useful for both tasks, improving the model's robustness.

[0021] In some embodiments, the method employs an end-to-end joint training approach during the training phase, wherein: during the training phase, the total loss function... loss from facial expression classification And occlusion segmentation loss Together they constitute the whole, and the calculation formula is as follows: ;in To balance the expression classification loss and the occlusion segmentation loss The hyperparameters; the occlusion segmentation loss Segmentation loss due to region occlusion and regional importance prediction loss Together constitute ,in To balance the hyperparameters.

[0022] This embodiment achieves a synergistic effect in multi-task learning. The facial expression recognition task and the occlusion segmentation task mutually reinforce each other: the segmentation task provides better feature focusing for recognition, while the high-level semantic feedback of the recognition task also helps improve segmentation accuracy (e.g., more accurately segmenting regions important for facial expression judgment). This embodiment simplifies the training process. Without the need for complex staged pre-training and fine-tuning of the two sub-networks, the optimized overall model can be obtained through a single end-to-end training, improving development efficiency.

[0023] In some embodiments, the total loss function Also includes: Feature contrast loss The feature contrast loss The final feature vector extracted from face images with different degrees of occlusion under the same expression category. Distance in feature space; the total loss function The calculation formula is: ,in For the feature contrast loss The weight hyperparameter.

[0024] This embodiment introduces feature contrast loss. It improves the model's feature learning ability and enhances its generalization ability and stability when dealing with unprecedented severe occlusion.

[0025] According to a second aspect of the present invention, an occluded facial expression recognition system is provided. The system includes: a data input module for acquiring a facial image to be recognized; and a dual-stream feature extraction module, which includes a facial expression stream submodule and an occlusion perception stream submodule connected in parallel, wherein the facial expression stream submodule is used to extract a global expression feature vector. The occlusion-aware flow submodule is used to extract occlusion-robust facial feature vectors. The dynamic fusion module includes a dynamic weight generator, used to generate weights based on the global facial expression feature vector. and the occlusion robust facial feature vector Generate dynamic fusion weights And based on the dynamic fusion weights For the global facial expression feature vector and the occlusion robust facial feature vector Perform weighted fusion to output the final feature vector. The expression classification module is used to classify the final feature vector. Perform classification and output the expression classification results.

[0026] This embodiment provides a specific system architecture that enables highly robust and adaptive occluded facial expression recognition, providing hardware carriers and system support for the practical application, deployment, and industrialization of this technology.

[0027] In some embodiments, the system further includes a model training module for performing the end-to-end joint training method.

[0028] This embodiment provides an integrated and complete model development and deployment solution, enabling the system not only to be used for inference and recognition, but also to have the ability to self-optimize and update, thereby improving the system's integrity and usability.

[0029] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0030] This invention abandons the traditional mindset of bypassing or repairing occlusions, and models and utilizes the occlusion information itself as an effective feature. It explicitly processes occlusions through a dedicated occlusion perception stream, and complements the global information of the facial expression stream. This allows the model to maintain a very high and stable recognition accuracy when faced with various random occlusions such as masks, glasses, and hands, effectively solving the problem of performance drop caused by occlusion in real-world scenarios in existing technologies.

[0031] The core dynamic feature fusion module of this invention can automatically and intelligently adjust the decision weights according to the specific content of each input image (such as the severity, location and type of occlusion); this mechanism makes the model no longer a rigid fixed pattern.

[0032] This invention enables weight sharing in shallow convolutional layers, reducing the total number of parameters and computational redundancy. The newly added occlusion segmentation network and dynamic weight generator are both lightweight designs. While introducing powerful new functions, the computational increment is controllable, ensuring the efficient operation of the model and making it easy to deploy on edge devices with limited computing resources, thus meeting the needs of real-time applications.

[0033] This invention provides not only classification results, but also a basis for decision-making. The mask output by the occlusion region segmentation network intuitively displays the occlusions located by the model, and the dynamic weights... The numerical representation reflects the model's level of trust in the two-stream information. This "white-box" decision-making mechanism enhances users' understanding and trust in the model's judgment results, aligns with the development direction of explainable artificial intelligence (XAI), and is crucial for applications in security-sensitive fields.

[0034] In the training phase, this invention integrates the occlusion segmentation task with the facial expression recognition task for collaborative optimization. These two tasks are inherently related in terms of low-level features (e.g., both require understanding facial structure). The segmentation task provides enhanced attention guidance for the recognition task, while the high-level semantic information of the recognition task also helps improve the accuracy of segmentation. The two mutually promote each other and jointly improve the overall performance of the model.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the specification, serve to explain the principles of the invention.

[0037] Figure 1 This is a flowchart of a method for recognizing occluded facial expressions based on a dynamic dual-stream network according to the present invention.

[0038] Figure 2 This is a system block diagram of an occluded facial expression recognition system based on a dynamic dual-stream network according to the present invention. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0042] Figure 1 This is a schematic flowchart illustrating an occluded facial expression recognition method based on a dynamic dual-stream network according to an embodiment of the present invention. The occluded facial expression recognition method based on a dynamic dual-stream network can be executed by a terminal. Terminals include, but are not limited to, mobile phones, tablets, wearable devices, sensors, IoT devices, and other communication devices.

[0043] See Figure 1 Occluded facial expression recognition methods based on dynamic dual-stream networks include:

[0044] S1. Obtain the face image to be identified;

[0045] S2. The face image is simultaneously input into a parallel facial expression stream and an occlusion detection stream; the facial expression stream is used to extract global expression feature vectors. The occlusion-aware stream is used to extract occlusion-robust facial feature vectors by segmenting the occluded region and guiding attention to the unoccluded facial region based on an attention mechanism. ;

[0046] S3, convert the global facial feature vector and the occlusion robust facial feature vector Input to the dynamic weight generator to generate dynamic fusion weights. ,in ;

[0047] S4. Use formula For the global facial expression feature vector and the occlusion robust facial feature vector Weighted fusion is performed to obtain the final feature vector. ;

[0048] S5. The final feature vector The input is fed into the classifier to obtain the expression classification results.

[0049] The following describes a specific implementation method.

[0050] Input preprocessing: The face image is read using the cv2.imread() function in the OpenCV library, and face detection and alignment are performed using the dlib library or MTCNN. The cropped face image is scaled to 224x224 pixels and normalized (e.g., normalizing pixel values ​​from [0, 255] to [0, 1] or using the mean and standard deviation of ImageNet).

[0051] Two-stream feature extraction: The preprocessed image is simultaneously input into two streams.

[0052] Facial Expression Stream (Stream A): Uses a ResNet-18 model pre-trained on large facial expression datasets such as FER2013 or AffectNet as the backbone network, removes its last classification layer, and outputs a 512-dimensional global feature vector. .

[0053] Occlusion-aware stream (Stream B): Using the same image input, extracting the image through the method in the subsequent steps of this claim. .

[0054] Dynamic feature fusion: and Two 512-dimensional feature vectors are concatenated into a 1024-dimensional vector, which is then input into a dynamic weight generator. This generator consists of a fully connected layer (output dimension 128, ReLU activation) and a fully connected layer (output dimension 1, Sigmoid activation), ultimately outputting a scalar weight value. According to the formula To integrate.

[0055] Facial expression classification: merging the feature vectors The input is fed into a classifier. This classifier consists of a fully connected layer (512-dimensional input, 7-dimensional output, corresponding to 7 basic expressions) and a Softmax function, which outputs the final class probability distribution.

[0056] In some embodiments, the occlusion-aware stream extracts the occlusion-robust facial feature vector. The methods include:

[0057] S21. The input face image is segmented using an occlusion region and an importance-aware network to generate a multi-channel mask image; wherein, the first channel of the multi-channel image is a binary mask used to identify the occlusion region. At least one additional channel in the multi-channel array is used to represent the importance weight of facial regions for expression recognition. ;

[0058] S22. According to the importance weight mask With the binary mask Generate soft attention maps for unoccluded regions ,in This indicates element-wise multiplication;

[0059] S23. Utilizing the soft attention map Modulate the multi-level intermediate feature maps in the occlusion sensing stream;

[0060] S24. Extract the occlusion-robust facial feature vector from the modulated feature map using a feature extraction network. .

[0061] The specific implementation methods of the above embodiments are as follows:

[0062] S21. Generate a multi-channel mask:

[0063] A fully convolutional network based on the U-Net architecture is used as the occlusion region and importance perception network. The encoder part of the network consists of four downsampling stages, each of which contains two 3x3 convolutional layers (followed by ReLU activation and batch normalization) and a 2x2 max pooling layer for downsampling. The decoder part consists of four symmetrical upsampling stages. Each stage first performs upsampling through a 2x2 transposed convolution, then concatenates the feature maps fused with the corresponding encoder stage through skip connections, and finally passes through two 3x3 convolutional layers (followed by ReLU and batch normalization). The final layer of the network uses a 1x1 convolutional layer to adjust the number of channels to 2 and uses the Sigmoid function as the activation function to output a 2-channel mask image M with the same size as the input image.

[0064] Among them, the first channel To mask the binary code, the ground truth labels during training are obtained manually at the pixel level and supervised by the binary cross-entropy loss function.

[0065] Second Channel The ground truth labels for the region importance weight map are automatically generated as follows: 68 facial keypoint coordinates are detected in the input image using the Dlib library or tools such as HRNet; based on prior knowledge of facial expression recognition, key regions such as eyes, eyebrows, and mouth are selected (e.g., points 18-27, 37-48, and 49-68 correspond to eyebrows, eyes, and mouth, respectively); a two-dimensional Gaussian heatmap with a standard deviation σ of 10 pixels is generated centered on these keypoint coordinates; finally, the heatmaps of each region are superimposed and normalized to the [0,1] interval to form the ground truth weights. This channel is supervised using the Mean-Squared Error-Loss function.

[0066] S22. Generate a soft attention map:

[0067] In the software implementation, a soft attention graph is generated through element-wise multiplication. Specifically, this involves weighting the importance graph. With the reverse occlusion mask (1- Multiply them together, that is The soft attention map generated by this operation enhances unoccluded facial regions that are crucial for expression recognition (such as the eyes and mouth), while suppressing feature responses from occluded regions and irrelevant areas such as the facial background.

[0068] S23, Multi-level Feature Modulation:

[0069] To guide feature extraction at multiple semantic levels, soft attention maps are used. The features are upsampled (using bilinear interpolation) to the same spatial size as the output feature maps of layer 2, layer 3, and layer 4 of the occlusion-aware stream backbone network (ResNet-18 in this example) (e.g., upsampled to 56x56, 28x28, and 14x14 pixels, respectively). Then, the upsampled attention map is multiplied element-wise with the feature map of the corresponding layer to achieve fine modulation of features at different scales. This multi-level attention mechanism ensures that the effective region is focused from low-level details to high-level semantic features.

[0070] S24, Robust Feature Extraction:

[0071] The modulated multi-layered feature maps are fed into subsequent layers of the backbone network for further processing. Finally, the features extracted by the network are reduced in dimensionality using a global average pooling layer, resulting in a 512-dimensional feature vector, which is the occlusion-resistant facial expression feature vector. .

[0072] In some embodiments, the occlusion region and importance-aware network includes a fully convolutional network based on an encoder-decoder structure.

[0073] The fully convolutional network in the encoder-decoder structure of this scheme is specifically implemented as U-Net. The encoder uses the first three blocks of ResNet-18 (a total of 4 convolutional stages, using the output of the 4th stage as the bottleneck feature), and the decoder correspondingly uses 4 upsampling blocks. Each block contains one bilinear upsampling, one skip connection corresponding to the output of the encoder stage, and two 3x3 convolutional layers. Finally, a 1x1 convolution reduces the number of channels to 1, and the final mask is output through the sigmoid function.

[0074] In some embodiments, the global facial feature vector and the occlusion robust facial feature vector Input to the dynamic weight generator to generate dynamic fusion weights. This includes: converting the global facial expression feature vector The occlusion robust facial feature vector and the occlusion statistical feature vector of the occlusion region extracted from the occlusion sensing stream. The dynamic weight generator is input, and the dynamic fusion weights are output after being calculated by a lightweight neural network module. .

[0075] The dynamic weight generator adaptively calculates the fusion weight α based on the occlusion status (such as occlusion area and location) of each input image, enabling the model to intelligently adjust its dependence on global features and occlusion-robust features. For example, when occlusion is severe (e.g., a mask covering the mouth), occlusion statistics (such as a large occlusion area and high coverage of key areas) will instruct the generator to output a smaller α (close to 0). Thus, during feature fusion, the model relies more on robust features extracted from the occlusion-aware flow. Because of robust characteristics Focusing on unoccluded areas is more reliable. However, when there is little or no occlusion, α approaches 1, and the model relies more on global facial expression features from the global expression stream output. This dynamic adaptability is something that cannot be achieved by fixed fusion models or single models that directly merge processes.

[0076] Dynamic weight generator requires global feature vectors The reason is that α's decision is not based solely on occlusion, but rather on a comparison between global facial information and occlusion.

[0077] For example, in cases of occlusion, if the global features contain strong facial signals (such as wide eyes indicating surprise), the dynamic weight generator may still assign a larger weight even with occlusion, and will not completely ignore the global features; however, in cases of severe occlusion and weak global features, the dynamic weight generator will assign a smaller weight, placing more trust in the robust features.

[0078] This comparison between global features and robust features ensures that the decision to determine the weight α is more intelligent: the dynamic weight generator can analyze the relationship between global features, robust features, and occlusion statistics to comprehensively judge and determine which of the three features, global features and robust features, is the more reliable facial expression feature vector.

[0079] Because the dynamic weight generator introduces occlusion statistics (such as occlusion area ratio, coverage ratio of key parts, etc.), the fusion process becomes interpretable. The numerical representation of weight α reflects the model's trust in the two-stream information, which aligns with the direction of interpretable artificial intelligence (XAI), whereas models that directly merge data would lose this transparency.

[0080] In some embodiments, the occlusion statistical feature vector The binary mask output by the occluded region segmentation network is determined using the following method. The data is analyzed, statistical features are extracted and encoded into a fixed-length vector as the occlusion statistical feature vector. The statistical features include at least one of the following: the percentage of the total area of ​​the occluded region, the coverage ratio of the occluded region in the eye, the coverage ratio of the occluded region in the mouth, and the coverage ratio of the occluded region in the eyebrow.

[0081] The occlusion statistical feature vector in this scheme The generation is achieved through the following specific steps:

[0082] Binarization: First, the occluded region is compared with the first channel mask output by the importance-aware network. (Its value range is [0,1]) Binarization is performed to clearly define the occluded area. A fixed threshold (preferably 0.5) is set to generate a binary mask. The calculation method is as follows:

[0083]

[0084] in, Represents the coordinates of a pixel.

[0085] Calculate the total occlusion ratio. ):

[0086] This statistic characterizes the degree of overall facial occlusion and is calculated using a binary mask. The number of pixels with a value of 1 Total number of pixels in the mask image The ratio:

[0087]

[0088] Calculate the occlusion ratio of key areas:

[0089] This set of statistics characterizes whether occlusions cover specific facial regions that are crucial for expression recognition. The specific steps are as follows:

[0090] Definition of key facial features: Utilizing predefined prior knowledge of facial structure and based on the annotation rules of the dlib68-point facial landmark model, the following three key regions are defined in the image pixel coordinate system:

[0091] Left-Eye-Region: A convex polygon whose vertices are formed by connecting the coordinates of key points [37,38,40,41].

[0092] Right-Eye-Region: A convex polygon whose vertices are formed by connecting the coordinates of key points [43,44,46,47].

[0093] Mouth-Region: A convex polygon whose vertices are formed by connecting the coordinates of key points [49,51,53,55,57,59].

[0094] Region pixel extraction: For each defined key region, the scan line algorithm is used to determine all pixels falling within the polygonal region, thereby obtaining the pixel set of that region.

[0095] Proportion Calculation: For each key area, calculate its occlusion proportion. The formula is:

[0096]

[0097] Wherein, the molecule is the total number of occluded pixels within the critical region (i.e., The denominator is the total number of pixels contained in the key region. .

[0098] Vector concatenation: Combining the four scalar statistics obtained from the above calculations—overall occlusion percentage. Percentage of left eye obstructed Percentage of right eye obstruction The proportion of mouth covering These are concatenated in this order to form a 4-dimensional real-valued vector, which is the interpretable occlusion statistical feature vector. .

[0099]

[0100] In some embodiments, the lightweight neural network module includes a multilayer perceptron model, wherein the generation of dynamic fusion weights... This includes: converting the global facial expression feature vector The occlusion robust facial feature vector and the occlusion statistical feature vector The concatenated vector is input into the multilayer perceptron model, and the dynamic fusion weights are output through the sigmoid activation function of the multilayer perceptron model. The dynamic fusion weight It is a scalar.

[0101] After that, (512 dimensions) (512 dimensions) and The three (4-dimensional) vectors are concatenated to form a new 1028-dimensional feature vector, which serves as the input to the dynamic weight generator.

[0102] The lightweight neural network module is a multilayer perceptron model, and its input is... , and The concatenated vectors are ultimately processed by the Sigmoid activation function to output scalar weights. .

[0103] In some embodiments, the facial expression stream and the occlusion perception stream share convolutional weights in a shallow network.

[0104] The specific implementation of this scheme is as follows: When defining the model, the encoder parts of the ResNet-18 in the facial expression stream (Stream A) and the segmentation network in the occlusion-aware stream (Stream B) (i.e., the encoder of U-Net) share the weights of the first three convolutional stages (e.g., layer1, layer2, and layer3 of ResNet-18). In PyTorch, this can be achieved by having the corresponding layers of the two network modules point to the same memory address, ensuring that the gradient will update the weights shared by the two streams simultaneously during training.

[0105] In some embodiments, the method employs an end-to-end joint training approach during the training phase, wherein: during the training phase, the total loss function... loss from facial expression classification And occlusion segmentation loss Together they constitute the whole, and the calculation formula is as follows: ;in To balance the expression classification loss and the occlusion segmentation loss The hyperparameters; the occlusion segmentation loss Segmentation loss due to region occlusion and regional importance prediction loss Together constitute ,in To balance the hyperparameters.

[0106] Total loss function It is the loss function for the entire end-to-end model, not just for a specific step or sub-module. The total loss function is used to jointly train the entire network. Among these, the expression classification loss... Corresponding to the expression classification module based on the final feature vector Facial expressions are categorized, and occlusion segmentation loss is applied. This corresponds to the occlusion segmentation loss used to segment occluded regions in the occlusion-aware stream. And the loss function used to determine the importance weights of facial regions for expression recognition. .

[0107] In practical applications, this scheme is trained using a dataset (such as RAF-OCC) that contains emoji labels and pixel-level occlusion annotations.

[0108] The total loss function is:

[0109] in:

[0110] The cross-entropy loss (corresponding to nn.CrossEntropyLoss in PyTorch) for the facial expression classification task is used to optimize the model's prediction accuracy for facial expression categories.

[0111] The binary cross-entropy loss (corresponding to nn.BCELoss in PyTorch) for occlusion segmentation only applies to the first channel of the mask image (i.e., the binary occlusion mask). This is used to optimize the model's segmentation performance in occluded areas.

[0112] The mean squared error loss (corresponding to nn.MSELoss in PyTorch) for the region importance prediction task only applies to the second channel of the mask image (i.e., the region importance weights). This is used to optimize the model's ability to predict the importance of key facial regions.

[0113] The loss balance hyperparameters can be empirically initialized to 1.0 and 0.5 respectively. In practical applications, they can be adjusted through cross-validation to achieve the optimal training effect.

[0114] The optimizer uses Adam with an initial learning rate of 1e-4. A dataset containing pixel-level occlusion annotations and facial expression labels (such as the RAF-OCC dataset) is used to meet the data requirements for multi-task joint training.

[0115] In some embodiments, the total loss function Also includes: Feature contrast loss The feature contrast loss The final feature vector extracted from face images with different degrees of occlusion under the same expression category. Distance in feature space; the total loss function The calculation formula is: ,in For the feature contrast loss The weight hyperparameter.

[0116] Specifically, within a training batch, for all sample pairs belonging to the same expression category... By calculating its final feature vector and The cosine similarity is used to force the feature similarity of similar samples to approach 1, thereby learning robust facial expression representations independent of occlusion. Its loss function formula is:

[0117]

[0118] The meanings of each parameter are as follows:

[0119] B: Number of samples in the training batch (Batch size).

[0120] : The set of other samples in the current batch that belong to the same expression category as sample i (i.e., the set of samples of the same category, excluding sample i itself).

[0121] Cosine similarity calculation function, used to measure the similarity between two final features. and The degree of similarity.

[0122] Temperature hyperparameter is used to adjust the "discrimination" of similarity to avoid gradient vanishing or training instability.

[0123] Corresponding to the embodiments of the occluded facial expression recognition method based on dynamic dual-stream network of the present invention, the present invention also provides embodiments of a corresponding occluded facial expression recognition system.

[0124] Please see Figure 2 , Figure 2 This is a system block diagram of an occluded facial expression recognition system based on a dynamic dual-stream network, according to one embodiment of the present invention. Figure 2 As shown, the occluded facial expression recognition system includes:

[0125] The data input module is used to acquire the face image to be identified;

[0126] The dual-stream feature extraction module includes a parallel-connected facial expression stream submodule and an occlusion-aware stream submodule. The facial expression stream submodule is used to extract global expression feature vectors. The occlusion-aware flow submodule is used to extract occlusion-robust facial feature vectors. ;

[0127] The dynamic fusion module includes a dynamic weight generator, used to generate weights based on the global facial expression feature vector. and the occlusion robust facial feature vector Generate dynamic fusion weights And based on the dynamic fusion weights For the global facial expression feature vector and the occlusion robust facial feature vector Perform weighted fusion to output the final feature vector. ;

[0128] The expression classification module is used to classify the final feature vector. Perform classification and output the expression classification results.

[0129] The above system can be deployed on a server equipped with a GPU, and the system includes the following modules:

[0130] Data input module: A Flask or Django-based Web API interface that receives images uploaded by the client;

[0131] Dual-stream feature extraction module: Loads pre-trained PyTorch or TensorFlow model files (.pt or .pb) into the server memory.

[0132] Dynamic fusion module and expression classification module: As part of the model computation graph, they are automatically invoked during inference;

[0133] The results output module returns the classification results (e.g., JSON format: {'expression': 'happy', 'confidence': 0.95}) to the client via the API.

[0134] This system module can be deployed in a containerized manner (using Docker) for easy scaling and management.

[0135] In some embodiments, the system further includes a model training module for performing end-to-end joint training.

[0136] The specific implementation of the model training module in this solution is an automated training pipeline based on PyTorch Lightning or TensorFlow Extended (TFX). This pipeline is integrated into the system, allowing administrators to automatically complete the entire process of data loading, model training, validation, and model deployment by uploading new labeled datasets, configuring hyperparameters (such as learning rate and λ value), and triggering training tasks, thereby achieving continuous learning and performance optimization of the system.

[0137] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. The invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0138] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The methods and apparatus provided in the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for recognizing occluded facial expressions based on a dynamic dual-stream network, characterized in that, The method includes: Obtain the face image to be identified; The facial image is simultaneously input into a parallel facial expression stream and an occlusion detection stream; the facial expression stream is used to extract global expression feature vectors. The occlusion-aware stream is used to extract occlusion-robust facial feature vectors by segmenting the occluded region and guiding attention to the unoccluded facial region based on an attention mechanism. The occlusion-aware stream extracts the occlusion-robust facial feature vector. The method includes: segmenting the input face image using an occlusion region and an importance-aware network to generate a multi-channel mask image; wherein, the first channel of the multi-channel image is a binary mask used to represent the occlusion region. At least one additional channel in the multi-channel array is used to represent a weighted mask representing the importance of facial regions for expression recognition. According to the importance weight mask With the binary mask Generate soft attention maps for unoccluded regions ,in This represents element-wise multiplication; utilizing the soft attention map. The multi-level intermediate feature maps in the occlusion-aware stream are modulated; the occlusion-robust facial feature vector is extracted from the modulated feature maps using a feature extraction network. ; The global facial expression feature vector The occlusion robust facial feature vector and the occlusion statistical feature vector of the occlusion region extracted from the occlusion sensing stream. Input to the dynamic weight generator to generate dynamic fusion weights. ,in ; Using formula For the global facial expression feature vector and the occlusion robust facial feature vector Weighted fusion is performed to obtain the final feature vector. ; The final feature vector The input is fed into the classifier to obtain the expression classification results.

2. The method according to claim 1, characterized in that, The occlusion region and importance-aware network includes a fully convolutional network based on an encoder-decoder structure.

3. The method according to claim 1, characterized in that, The generation of dynamic fusion weights ,include: The global facial expression feature vector The occlusion robust facial feature vector and the occlusion statistical feature vector The dynamic weight generator is input, and the dynamic fusion weights are output after being calculated by a lightweight neural network module. .

4. The method according to claim 1, characterized in that, The occlusion statistical feature vector The binary mask output by the occluded region segmentation network is determined using the following method. The data is analyzed, statistical features are extracted and encoded into a fixed-length vector as the occlusion statistical feature vector. The statistical features include at least one of the following: the percentage of the total area of ​​the occluded region, the coverage ratio of the occluded region in the eye, the coverage ratio of the occluded region in the mouth, and the coverage ratio of the occluded region in the eyebrow.

5. The method according to claim 3, characterized in that, The lightweight neural network module includes a multilayer perceptron model, wherein the generation of dynamic fusion weights... ,include: The global facial expression feature vector The occlusion robust facial feature vector and the occlusion statistical feature vector The concatenated vector is input into the multilayer perceptron model, and the dynamic fusion weights are output through the sigmoid activation function of the multilayer perceptron model. The dynamic fusion weight It is a scalar.

6. The method according to claim 1, characterized in that, The facial expression stream and the occlusion perception stream share convolutional weights in the shallow network.

7. The method according to claim 2, characterized in that, The method employs an end-to-end joint training approach during the training phase, wherein: During the training phase, the total loss function loss from facial expression classification And occlusion segmentation loss Together they constitute the whole, and the calculation formula is as follows: ;in To balance the expression classification loss and the occlusion segmentation loss hyperparameters; The occlusion segmentation loss Segmentation loss due to region occlusion and regional importance prediction loss Together constitute ,in To balance the hyperparameters.

8. The method according to claim 7, characterized in that, The total loss function Also includes: Feature contrast loss The feature contrast loss The final feature vector extracted from face images with different degrees of occlusion under the same expression category. Distance in feature space; The total loss function The calculation formula is: ,in For the feature contrast loss The weight hyperparameter.

9. A facial expression recognition system with occlusion, characterized in that, include: The data input module is used to acquire the face image to be identified; The dual-stream feature extraction module includes a parallel-connected facial expression stream submodule and an occlusion-aware stream submodule. The facial expression stream submodule is used to extract global expression feature vectors. The occlusion-aware flow submodule is used to extract occlusion-robust facial feature vectors. The occlusion-aware flow submodule is used to extract occlusion-robust facial feature vectors. The method includes: segmenting the input face image using an occlusion region and an importance-aware network to generate a multi-channel mask image; wherein, the first channel of the multi-channel image is a binary mask used to represent the occlusion region. At least one additional channel in the multi-channel array is used to represent a weighted mask representing the importance of facial regions for expression recognition. According to the importance weight mask With the binary mask Generate soft attention maps for unoccluded regions ,in This represents element-wise multiplication; utilizing the soft attention map. The multi-level intermediate feature maps in the occlusion-aware stream are modulated; the occlusion-robust facial feature vector is extracted from the modulated feature maps using a feature extraction network. ; The dynamic fusion module includes a dynamic weight generator, used to generate weights based on the global facial expression feature vector. The occlusion robust facial feature vector and the occlusion statistical feature vector of the occlusion region extracted from the occlusion sensing stream. Generate dynamic fusion weights And based on the dynamic fusion weights For the global facial expression feature vector and the occlusion robust facial feature vector Perform weighted fusion to output the final feature vector. ; The expression classification module is used to classify the final feature vector. Perform classification and output the expression classification results.

Images