Multi-modal fusion expression recognition method based on adaptive occlusion perception

Abstract

The invention discloses a shielded facial expression recognition method based on a self-adaptive gated double-flow hybrid expert network. The method comprises the following steps: S1, obtaining a to-be-analyzed facial image and carrying out the preprocessing of the to-be-analyzed facial image; s2, inputting the preprocessed face image into a shielding detector, and extracting a shielding semantic feature vector and a shielding score; s3, respectively inputting the face image into a global feature expert network and a local feature expert network for feature extraction; s4, according to the shielding semantic features and the shielding scores, generating corresponding expert weights through an adaptive gating network; and S5, performing weighted fusion on the global features and the local features according to the expert weights to obtain an expression recognition result. According to the method, the feature fusion strategy can be dynamically adjusted according to different shielding degrees, and the accuracy and robustness of facial expression recognition in a complex shielding scene are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, pattern recognition, and affective computing, specifically to a method for occluded facial expression recognition based on an adaptive gated dual-stream hybrid expert network, which is particularly suitable for the difficulty of facial expression recognition in real-world scenarios caused by wearing masks, hands, or objects obscuring the face. Background Technology

[0002] With the rapid development of fields such as human-computer interaction, intelligent driving assistance systems, mental health monitoring, and social robots, facial expression recognition, as one of the core tasks in the field of computer vision, is increasingly demonstrating its practical application value. Traditional expression recognition methods mainly rely on manually extracted geometric or texture features, but their robustness often fails to meet practical needs when facing uncontrolled environments such as changes in lighting, pose shifts, and facial occlusion.

[0003] In recent years, the rise of deep learning technology has fundamentally changed the research paradigm of FER (Feature Extraction). Convolutional Neural Networks (CNNs), with their powerful local feature extraction capabilities, have become the mainstream architecture in this field. For example, models based on backbone networks such as ResNet and VGG have achieved extremely high recognition accuracy on controlled datasets. Furthermore, with the introduction of attention mechanisms and visual Transformers, the ability of models to capture global facial semantic information and long-range dependencies has been further enhanced. Hybrid architectures such as POSTER, by combining the local perception advantages of CNNs with the global modeling advantages of Transformers, have broken performance records on public benchmark datasets.

[0004] However, during the process of realizing this invention, the inventors discovered through in-depth research that existing technologies still have significant technical bottlenecks and defects when facing real-world complex scenarios—especially scenarios with facial occlusion: Existing models lack explicit awareness and adaptive capabilities regarding the degree of occlusion: Currently, mainstream facial expression recognition systems typically employ a "one-size-fits-all" processing logic, meaning that regardless of whether the input image is clear and unobstructed, or whether it is covered by a mask, sunglasses, or hands, it is processed using the same set of fixed network weights.

[0005] For clear images, complex anti-occlusion mechanisms may introduce unnecessary noise and reduce the recognition limit.

[0006] For occluded images, standard models often fail to distinguish between "effective regions" and "occlusion noise," causing convolutional kernels to be affected by the texture of the occluded objects when extracting features, resulting in incorrect classification results. Existing technologies lack an adaptive mechanism that can first "determine the degree of occlusion" and then "dynamically adjust the strategy."

[0007] A single network architecture has limitations when dealing with different types of occlusion: Different neural network architectures have different inductive biases, making it difficult to maintain optimality under all occlusion conditions. CNNs excel at extracting local textures, but they are prone to failure when severe occlusion causes the complete loss of key local features due to a lack of global contextual reasoning capabilities.

[0008] Transformer: Although it has a global receptive field and can infer overall emotion from unoccluded areas, its computational complexity increases quadratically with image resolution. It has high inference latency and is prone to overfitting on small datasets.

[0009] State-space model: As an emerging architecture, it possesses linear computational complexity ( With its powerful long sequence modeling capabilities, it is theoretically well-suited for processing visual information streams interrupted by occlusion, but research on its application to facial expression recognition and specifically targeting the occlusion problem is still lacking.

[0010] The shortcomings of local feature decoupling and reconstruction techniques: To address the occlusion problem, some existing studies have attempted to use feature decoupling or completion techniques. However, generative completion methods are computationally expensive, and the generated facial details often contain artifacts, potentially misleading the classifier. While attention-based weighted methods can suppress the weights of occluded regions, the remaining effective features are often insufficient to support high-precision classification when there is large-area occlusion. Therefore, a mechanism capable of mining deep, latent semantic features is urgently needed.

[0011] The singularity of data augmentation strategies: The occlusion patterns generated by random erasure or mesh occlusion commonly used in current FER training are rather mechanical and fail to simulate the complex occlusions with semantic information in real life. This difference between the training data distribution and the actual test distribution leads to a significant performance drop in the model during actual deployment.

[0012] In summary, there is an urgent need for an adaptive multimodal fusion expression recognition method that can intelligently perceive the occlusion state of the input image and integrate the local fineness of CNNs, the global reasoning ability of Transformer / Mamba, and the anti-interference advantages of feature decoupling networks, in order to break through the performance ceiling of current technology in complex occlusion scenarios. Summary of the Invention

[0013] The purpose of this invention is to provide an occluded facial expression recognition method based on an adaptive gated dual-stream hybrid expert network. By constructing a dual-stream hybrid expert architecture and introducing an occlusion-aware gating mechanism, dynamic complementarity between global and local features is achieved, thereby improving the recognition accuracy in occluded environments.

[0014] Includes the following steps: S1: Obtain the sequence of face images to be analyzed, and preprocess the face images; S2: Input the preprocessed face image into the pre-built occlusion detector model, extract and output the corresponding occlusion semantic feature vector. and scalar occlusion score The occlusion detector model is trained based on a hybrid occlusion enhancement strategy to obtain the ability to continuously perceive the degree of occlusion. S3: The preprocessed face images are input into a parallel dual-stream expert network for feature extraction and classification prediction, including: The first stream consists of a global feature expert network, which takes a preprocessed face image as input. Based on a visual state-space model architecture, the system serializes intermediate feature maps using a two-dimensional multi-directional selective scanning mechanism. It then combines grouping parameter broadcast correction and gated modulation with recursive state-space computation to extract global feature representations with long-range dependencies. After aligning and aggregating features from different scanning directions, it generates classification outputs corresponding to global feature experts. ; The second stream is a local feature expert network. It takes the face image or its resampled multi-scale feature representation as input and constructs a network architecture based on a combination of feature decomposition and feature reconstruction. Specifically, a feature decomposition network containing multiple parallel patch gating units performs saliency selection and dimensionality reduction on local patch features. Then, a feature reconstruction network containing intra-frame relation modules and inter-frame relation modules performs joint modeling and residual fusion on multiple local feature branches to form a local feature representation. Finally, it generates the classification output corresponding to the local feature expert. ; S4: Construct an occlusion-aware adaptive gating network Using the occlusion semantic feature vector and occlusion score obtained in step S2 as gating input, an expert weight vector for the current input image is generated through a calculation method that includes occlusion prior modeling, semantic correction, and weight modulation mechanisms. The expert weights mentioned therein change dynamically with the degree of occlusion, and are used to characterize the contribution ratio of global feature experts and local feature experts under different occlusion states. S5: Based on the expert weight vector, output the classification of global feature experts. Classification output from local feature experts Weighted fusion is performed to obtain the final expression classification prediction result: The occlusion perception gating mechanism enables adaptive coordination between global semantic modeling capabilities and local fine-grained discrimination capabilities.

[0015] The hybrid occlusion enhancement strategy in step S2 is trained, including: Let the original training image be denoted as Through random occlusion operator Generate occlusion enhancement image : Where parameters Indicates the occlusion type and occlusion area parameters; The occlusion operator includes at least the following three forms: (1) Mask occlusion: Generate a rectangular occlusion area in the lower half of the image. ,satisfy in This is the scaling factor for the lower half of the region; (2) Hand occlusion: Generate an elliptical occlusion area in the image. Its center position and major and minor axis parameters are randomly sampled according to a preset probability distribution, and mouth occlusion and eye occlusion are distinguished according to the occlusion position; (3) General Occlusion: Generate random noise occlusion areas at any location in the image. Its pixel values ​​follow a uniform or Gaussian distribution; Calculate the occlusion ratio based on the ratio of the occluded area to the total image area: And based on the stated occlusion ratio The training samples were divided into four states: no occlusion, moderate occlusion, palm occlusion, and severe occlusion. Occlusion detector based on occlusion ratio As a regression supervision signal, training is performed by minimizing the regression loss function: in This represents the occlusion score predicted by the model.

[0016] The global feature expert network in step S3 employs an improved selective scanning algorithm, specifically including: The intermediate feature map output by the pre-coding module of the global feature expert network is denoted as...

[0017] Where Batch is the batch size and Dim is the feature channel dimension; The feature map The sequence is unfolded into a one-dimensional sequence along four directions: horizontal, vertical, and reverse, yielding the sequence features. ,in The length of the expanded sequence; Grouping parameters for constructing state-space models and The parameters mentioned above are used for input and output mapping in the hidden state, and the number of channels is [number missing]. ,and To achieve parameter sharing; When the feature channel dimension is detected Can be During division, a channel broadcast operation is performed on the grouping parameters, which copies the parameters along the channel dimension, so that they are distributed from... Expand to So that the grouping parameters act on the entire feature channel, wherein Represents the internal state dimension of the state space; For continuous state space parameters and Discretization is performed to obtain discrete state parameters, and state-space recursive calculation is performed for each expansion direction. The hidden state satisfies: in This indicates the hidden state corresponding to the current scan direction and the current time step; Calculate intermediate output features based on the hidden states: And introduce a gating coefficient Adaptive modulation is applied to the output features; The modulation output features from different scanning directions are aggregated to form a global feature representation, and a global classification output is obtained through linear mapping.

[0018] The local feature expert network in step S3 includes a feature decomposition network and a feature reconstruction network for generating local classification outputs, and its methods include: The input face image or its multi-scale feature representation is divided into multiple local patch regions, and feature encoding is performed on each local patch region to obtain a local patch feature set. in Indicates the patch index. Indicates a time step; The feature decomposition network consists of multiple parallel patch gating units. Each patch gating unit assigns saliency weights to the corresponding local patch features through an attention subnetwork. The local patch features are then mapped and dimensionality-reduced using a feature transformation subnet to obtain multiple local feature branches: in Represents the feature mapping function; The feature reconstruction network includes an intra-frame relation module and an inter-frame relation module, wherein: The intra-frame relation module is used to model the spatial correlation between different local feature branches within the same time step. It constructs an intra-frame relation matrix by calculating the weighted correlation between any two local feature branches. And based on the intra-frame relation matrix, the local features are enhanced to improve their relationships: This is to characterize the complementary and synergistic relationships between different local regions; the inter-frame relationship module is used to model the temporal correlation of local feature branches between adjacent time steps. It performs differential modeling of the local features of the current time step and the previous time step, and introduces a gating coefficient to control the fusion intensity of temporal information. in These are time-series gating coefficients that are dynamically generated as the features change. Local features are jointly modeled using the intra-frame relation module and the inter-frame relation module, and the reconstructed local features are fused using residual connections to form a local feature representation. Based on the local feature representation, feature aggregation and classification mapping are performed to generate local classification outputs corresponding to local feature experts, which are then fused with the outputs of global feature experts.

[0019] The adaptive gating network in step S4 employs a weight generation method of "prior weights + correction terms + temperature control," specifically including: Let the occlusion semantic feature vector be... The occlusion score is First, the prior terms for expert weights are generated from the occlusion scores: in For the Sigmoid function, The slope coefficient, The switching threshold; Then, a correction term for the prior term is generated from the occlusion semantic features: in For feature mapping function, These are learnable parameters; The global expert weights are obtained by fusing prior terms and correction terms, and the local expert weights are obtained through complementary constraints. Alternatively, the two can be normalized to form a weight vector; The weight distribution is modulated by introducing a temperature coefficient that varies with the degree of occlusion: in Follow Adaptive changes are made to achieve smoothness and stability in weight switching; The final fusion output is: in and It dynamically changes according to the degree of occlusion to achieve complementary collaboration between global experts and local experts.

[0020] The technical effects to be achieved by the embodiments of the present invention are as follows: Strong anti-occlusion capability: Through a specially designed occlusion detector and adaptive gating, the system can rely on the global advantages of VMamba when there is no occlusion, and automatically switch to focus on the unoccluded local features of FOPM or the context reasoning capabilities of VMamba when there is partial occlusion.

[0021] High computational efficiency: VMamba has linear computational complexity, which is more suitable for real-time processing compared to the quadratic complexity of Vision Transformer; and this invention removes the redundant POSTER module, further lightweighting the model.

[0022] Robust technical implementation: A broadcast correction mechanism for grouping parameters was added to the Selective Scan operator, which solved the runtime error caused by dimension mismatch and enhanced the compatibility and stability of the model. Attached Figure Description

[0023] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0024] Figure 1 This is a flowchart illustrating the overall process architecture of the method of the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of an adaptive gating network.

[0026] Figure 3 A schematic diagram illustrating the principle and parameter broadcasting of the improved selective scanning mechanism.

[0027] Figure 4 This is a graph showing the weight variation trend of a dual-stream expert network under different degrees of occlusion. Detailed Implementation

[0028] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0029] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0030] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0031] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0032] In the specification, when an element is described as being "on" another element, "connected to" or "bonded to" another element, the element may be directly "on" another element, directly "connected to" or "bonded to" another element, or there may be one or more other elements in between. Conversely, when an element is described as being "directly on" another element, "directly connected to" or "directly bonded to" another element, there may be no other elements in between.

[0033] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0034] Unless otherwise defined, all terms used herein shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0035] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0036] Example 1: System Overall Architecture like Figure 1 As shown, this system adopts a dual-stream hybrid expert architecture. (Input image) .

[0037] Occlusion detector branch: Input Output occlusion features And occlusion score .

[0038] Global expert branch: Input After patch embedding and stacked VSS blocks, the global classification logits are output. .

[0039] Local expert branch: Input upsampled After passing through the FDN and FRN modules, the local classification Logits are output. .

[0040] Fusion Module: Gated Network Output weights final result .

[0041] Example 2: Improved Occlusion Enhancement and Detection To train a highly robust occlusion detector, this invention designs the OcclusionAugmentation class.

[0042] Unlike simple random erasure, this embodiment achieves semantic-level occlusion simulation: Hand occlusion: Generates an elliptical shape, with the color taken from the skin tone range. The position tends to cover the mouth or eyes.

[0043] Mask coverage: Restricted to the lower half of the face, simulating the geometric constraints of wearing a mask.

[0044] Forced Distribution Strategy: When training ImprovedOcclusionDetector, instead of relying entirely on random probability, samples are forced to be distributed proportionally according to [no occlusion, mask / sunglasses, hands, severe occlusion] to avoid overfitting the model to a particular type of occlusion. The detector uses a ResNet18 backbone, with the output layer changed to a Sigmoid-activated regression head, and supervised training is performed using MSELoss.

[0045] Example 3: Improved Selective Scan Core Operator In this embodiment, the global feature expert network is constructed based on a visual state space model, and its core computation process is implemented through a selective scanning operator. Specifically, the selective scanning operator performs recursive state space computation along the feature sequence dimension to model long-range dependencies of the unfolded visual features.

[0046] In the standard state-space model, the input feature sequence is represented as: in For batch size, For feature channel dimension, The sequence length; the corresponding state transition parameters. Having dimensions in For the hidden state dimension.

[0047] However, when designing lightweight solutions for vision tasks, input mapping parameters are used to reduce parameter size and achieve cross-channel parameter sharing. Output mapping parameters It is set to grouped form, and its dimensions are represented as in Let be the number of group channels, and satisfy . In this setting, each set of parameters corresponds to multiple feature channels.

[0048] If we directly substitute the above grouping parameters into the state-space recursive calculation formula and output calculation formula This will cause the parameter channel dimension to be inconsistent with the input feature channel dimension, thus making it impossible to complete the channel-by-channel state update and feature mapping, resulting in a problem of mismatch in computational dimensions.

[0049] To address the aforementioned problems, this invention introduces a dynamic channel broadcast correction mechanism into the selective scanning operator. Specifically, when a feature channel dimension is detected... Number of channels that can be grouped When divisible, calculate the repetition factor. And grouping parameters along the channel dimension and Perform a copy extension so that it is from Expand to This allows each feature channel to obtain corresponding input mapping parameters and output mapping parameters.

[0050] Through the aforementioned channel broadcast correction operation, the discretized state-space computational term... It can correctly perform element-wise operations on the full feature channel dimension, enabling state space recursive computation to achieve stable state updates for all feature channels while maintaining the advantages of grouped parameter sharing and model lightweighting.

[0051] Therefore, the dynamic broadcast correction mechanism described in this embodiment effectively solves the dimensionality mismatch problem of selective scanning operators in the process of visual feature modeling under grouping parameter settings, and enhances the applicability and stability of the state space model in high-dimensional visual feature scenarios.

[0052] Example 4: Adaptive Gating and Joint Training The design of the gating network is the key to achieving "adaptiveness" in this invention.

[0053] Structurally, ExpertGatingNetwork receives two parts of input: Explicit score: A 0-1 scalar predicted by the occlusion detector, which directly reflects the degree of occlusion.

[0054] Implicit features: 512-dimensional deep features extracted by the occlusion detector, which include information on the location and type of occlusion.

[0055] This design allows the gating network to know not only "how much occlusion there is," but also "where the occlusion is located," thus enabling it to allocate weights more intelligently. For example, if the mouth is blocked, the gating network will tend to increase the weight of VMamba.

[0056] The training process employs a "freeze-fine-tuning" strategy: Load weights: Load the pre-trained VMamba, FOPM, and occlusion detector weights.

[0057] Freeze parameters: Set the requires_grad parameter of the above three models to False.

[0058] Training the gating network: Only update the parameters of the ExpertGatingNetwork. Train using a mixed dataset, forcing the gating network to learn the optimal weight allocation under different occlusions.

[0059] TTA Prediction: During the inference phase, test-time enhancement is introduced, which predicts the original image and the horizontally flipped image separately and then averages the results to further improve stability.

[0060] Example 5: Specific configuration and strategies for model training To ensure the convergence and generalization ability of the model, this embodiment details the key hyperparameter settings during the training process: Optimizer Grouping Configuration: A grouping learning rate strategy is adopted to address the different convergence speeds of various components in the two-stream architecture. Using the Adam optimizer, the learning rate of the adaptive gating network is set to 5e-5 to facilitate its rapid adaptation to different occlusion modes; while the learning rates of the pre-trained occlusion detector, global expert, and local expert are set to an extremely low 5e-6, only fine-tuned, to prevent damage to existing feature extraction capabilities.

[0061] Loss function design: A label-smoothed cross-entropy loss function is used, with a smoothing factor set to 0.1. This setting not only helps alleviate overfitting but also improves the model's robustness in the presence of occlusion noise.

[0062] Learning rate scheduling: A cosine annealing strategy is used to dynamically adjust the learning rate to ensure that the model can escape local optima in the later stages of training and achieve a better convergence state.

[0063] Random seed fixation: To ensure the reproducibility of the experiment, the random seeds for NumPy, PyTorch, and CUDA are strictly fixed before training begins.

[0064] Example 6: Verification and Analysis of Adaptive Gating Mechanism To verify the effectiveness of the "adaptive" mechanism proposed in this invention, this embodiment performs statistical analysis on the gating behavior on the validation set: Data collection: Run the model on the validation set to collect the predicted occlusion score and weights of the gating network output for each image.

[0065] Correlation analysis: Pearson correlation coefficient was calculated. The results showed that VMamba weights were significantly negatively correlated with occlusion scores. ), and the weight of FOPM is significantly positively correlated with the occlusion score ( This indicates that the gated network has successfully learned the logic that "the heavier the occlusion, the more it relies on local details".

[0066] Crossover Phenomenon: A graph showing the relationship between occlusion score and weight contribution reveals a clear "X"-shaped crossover curve. When the occlusion score is below a certain threshold, VMamba weights dominate; when the occlusion score is above this threshold, FOPM weights take over. This phenomenon intuitively demonstrates that the system can intelligently switch between global dependencies and local details, validating the complementarity of the two-stream architecture.

[0067] Corresponding to the above method, this application embodiment also provides a computer-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and run by a processor, the machine-executable instructions cause the processor to perform the steps of the above method.

[0068] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0071] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0072] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A method for occluded facial expression recognition based on an adaptive gated dual-stream hybrid expert network, characterized in that, Includes the following steps: S1: Obtain the sequence of face images to be analyzed and perform preprocessing; S2: Input the preprocessed image x into the pre-built occlusion detector model, extract and output the deep occlusion semantic feature vector of the image. and scalar occlusion fraction The occlusion detector model is trained based on a hybrid occlusion enhancement strategy. S3: Input the images into a parallel two-stream expert network for feature extraction. The first stream is a global feature expert network. It takes a preprocessed image x as input and, based on a visual state-space model architecture, uses a two-dimensional selective scanning mechanism to extract global feature logs with long-range dependencies. After patch embedding and stacked VSS blocks, it outputs a global classification. ; The second stream is a local feature expert network, which takes the preprocessed image x as input and upsamples it. Based on the feature decomposition and reconstruction network architecture, fine-grained logarithms of key local regions are extracted using patch gating units. After passing through FDN and FRN modules, local classification is output. ; S4: Constructing an adaptive gating network The system receives the occlusion semantic feature vector and occlusion score extracted in step S2, generates an expert weight vector for the current image through multilayer perceptron mapping, and outputs the weights. ; S5: Based on the expert weight vector, perform a weighted fusion of the outputs of the global feature expert and the local feature expert: This yields the final facial expression classification prediction results.

2. The method according to claim 1, characterized in that, The hybrid occlusion enhancement strategy in step S2 is trained, including: Let the original training image be denoted as Through random occlusion operator Generate occlusion enhancement image : Where parameters Indicates the occlusion type and occlusion area parameters; The occlusion operator includes at least the following three forms: (1) Mask occlusion: Generate a rectangular occlusion area in the lower half of the image. ,satisfy in This is the scaling factor for the lower half of the region; (2) Hand occlusion: Generate an elliptical occlusion area in the image. Its center position and major and minor axis parameters are randomly sampled according to a preset probability distribution, and mouth occlusion and eye occlusion are distinguished according to the occlusion position; (3) General Occlusion: Generate random noise occlusion areas at any location in the image. Its pixel values ​​follow a uniform or Gaussian distribution; Calculate the occlusion ratio based on the ratio of the occluded area to the total image area: And based on the stated occlusion ratio The training samples were divided into four states: no occlusion, moderate occlusion, palm occlusion, and severe occlusion. Occlusion detector based on occlusion ratio As a regression supervision signal, training is performed by minimizing the regression loss function: in This represents the occlusion score predicted by the model.

3. The method according to claim 1, characterized in that, The global feature expert network in step S3 employs an improved selective scanning algorithm, specifically including: The intermediate feature map output by the pre-coding module of the global feature expert network is denoted as... Where Batch is the batch size and Dim is the feature channel dimension; The feature map The sequence is unfolded into a one-dimensional sequence along four directions: horizontal, vertical, and reverse, yielding the sequence features. ,in The length of the expanded sequence; Grouping parameters for constructing state-space models and The parameters mentioned above are used for input and output mapping in the hidden state, and the number of channels is [number missing]. ,and To achieve parameter sharing; When the feature channel dimension is detected Can be During division, a channel broadcast operation is performed on the grouping parameters, which copies the parameters along the channel dimension, so that they are distributed from... Expand to So that the grouping parameters act on the entire feature channel, wherein Represents the internal state dimension of the state space; For continuous state space parameters and Discretization is performed to obtain discrete state parameters, and state-space recursive calculation is performed for each expansion direction. The hidden state satisfies: in This indicates the hidden state corresponding to the current scan direction and the current time step; Calculate intermediate output features based on the hidden states: And introduce a gating coefficient Adaptive modulation is applied to the output features; The modulation output features from different scanning directions are aggregated to form a global feature representation, and a global classification output is obtained through linear mapping.

4. The method according to claim 1, characterized in that, The local feature expert network in step S3 includes a feature decomposition network and a feature reconstruction network for generating local classification outputs, and its methods include: The input face image or its multi-scale feature representation is divided into multiple local patch regions, and feature encoding is performed on each local patch region to obtain a local patch feature set. in Indicates the patch index. Indicates a time step; The feature decomposition network consists of multiple parallel patch gating units. Each patch gating unit assigns saliency weights to the corresponding local patch features through an attention subnetwork. The local patch features are then mapped and dimensionality-reduced using a feature transformation subnet to obtain multiple local feature branches: in Represents the feature mapping function; The feature reconstruction network includes an intra-frame relation module and an inter-frame relation module, wherein: The intra-frame relation module is used to model the spatial correlation between different local feature branches within the same time step. It constructs an intra-frame relation matrix by calculating the weighted correlation between any two local feature branches. And based on the intra-frame relation matrix, the local features are enhanced to improve their relationships: This is to characterize the complementary and synergistic relationships between different local regions; the inter-frame relationship module is used to model the temporal correlation of local feature branches between adjacent time steps. It performs differential modeling of the local features of the current time step and the previous time step, and introduces a gating coefficient to control the fusion intensity of temporal information. in These are time-series gating coefficients that are dynamically generated as the features change. Local features are jointly modeled using the intra-frame relation module and the inter-frame relation module, and the reconstructed local features are fused using residual connections to form a local feature representation. Based on the local feature representation, feature aggregation and classification mapping are performed to generate local classification outputs corresponding to local feature experts, which are then fused with the outputs of global feature experts.

5. The method according to claim 1, characterized in that, The adaptive gating network in step S4 employs a weight generation method of "prior weights + correction terms + temperature control," specifically including: Let the occlusion semantic feature vector be... The occlusion score is First, the prior terms for expert weights are generated from the occlusion scores: in For the Sigmoid function, The slope coefficient, The switching threshold; Then, a correction term for the prior term is generated from the occlusion semantic features: in For feature mapping function, These are learnable parameters; The global expert weights are obtained by fusing prior terms and correction terms, and the local expert weights are obtained through complementary constraints. Alternatively, the two can be normalized to form a weight vector; The weight distribution is modulated by introducing a temperature coefficient that varies with the degree of occlusion: in Follow Adaptive changes are made to achieve smoothness and stability in weight switching; The final fusion output is: in and It dynamically changes according to the degree of occlusion to achieve complementary collaboration between global experts and local experts.

6. The method according to claim 1, characterized in that, The model training of the method adopts a phased joint fine-tuning strategy: Phase 1: Pre-train the occlusion detector, global feature expert network, and local feature expert network independently, and save the optimal weights; The second stage involves loading the optimal weights, freezing all parameters of the global and local feature expert networks, and training only the adaptive gating network; or setting a high learning rate for the gating network, setting a fine-tuned learning rate for the expert networks, and applying random occlusion enhancement to the training samples with a 50% probability to improve the gating network's sensitivity to occlusion changes.

7. A facial expression recognition system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

Images