A deep fusion enhanced bidirectional Mamba facial expression recognition method

Abstract

The application discloses a kind of bidirectional Mamba facial expression recognition methods based on depth fusion enhancement, belong to computer vision and emotional computing field, with ResNet18 as backbone network, construct parallel double branch after layer3, first branch carries out multi-scale strip gate attention enhancement and complex frequency domain adaptive enhancement in turn to middle layer space feature, second branch carries out independent bidirectional Mamba sequence long-range dependence modeling, and the output of two branches is fused to obtain enhanced feature;After layer4, the fusion enhanced feature is sequentially subjected to two-dimensional space bidirectional Mamba aggregation and sparse attention refining, to obtain high-level semantic features;High-level semantic features are classified and predicted, and the model is trained using focal loss with fused label smoothing.The application can effectively improve the precision and robustness of facial expression recognition by spatial and frequency domain feature collaborative enhancement, two-dimensional space full-direction long-range dependence modeling and key area focusing.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and emotion computing technology, and in particular relates to a bidirectional Mamba facial expression recognition method based on deep fusion enhancement. Background Technology

[0002] Facial expression recognition, as an important research direction in computer vision and affective computing, aims to automatically analyze and identify the emotional state in facial images through deep learning models. Its technological achievements have been widely applied in practical scenarios such as intelligent human-computer interaction, assisted driving safety monitoring, smart healthcare emotion assessment, and security monitoring. Among existing technologies, facial expression recognition methods based on Convolutional Neural Networks (CNNs) can effectively extract local texture and edge information in the spatial domain of images, achieving good recognition performance in various environments. Methods based on the Transformer architecture achieve global context modeling through a multi-head self-attention mechanism, effectively capturing long-range dependencies between different facial regions. Methods based on state-space models (such as Mamba), with their linear complexity sequence modeling capabilities, significantly reduce computational overhead while maintaining global perception capabilities, showing promising application prospects. Furthermore, sparse attention mechanisms effectively reduce redundant computation and improve the model's ability to focus on discriminative features by selecting key attention connections. Frequency domain feature enhancement techniques transform images to the frequency domain through Fourier transform, enabling the extraction of texture details and structural information that are difficult to obtain directly in the spatial domain. These technologies provide valuable technical support for facial expression recognition from different perspectives.

[0003] However, the aforementioned existing technologies still have many shortcomings. First, although traditional convolutional neural networks can extract local spatial features, they are limited by fixed local receptive fields, making it difficult to fully model the long-range dependencies of facial muscle movements in the global context, and limiting their ability to discriminate subtle differences in expressions such as fear and surprise. While the Transformer architecture possesses global modeling capabilities, the secondary computational complexity of its self-attention mechanism leads to a significant increase in the number of parameters and computational overhead, and it lacks sufficient refinement in characterizing local fine-grained features. Second, existing methods do not fully utilize frequency domain information. Changes in facial expressions are simultaneously manifested as coordinated changes in high-frequency details (such as wrinkles and trembling at the corners of the mouth) and low-frequency structures (such as facial contours) in the frequency domain. Conventional spatial domain convolution operations cannot achieve adaptive separation and efficient utilization of multi-scale frequency domain components, thus restricting the model's feature representation capabilities under complex lighting conditions. Third, existing methods generally lack an effective collaborative mechanism between spatial domain feature extraction, frequency domain feature enhancement, and global dependency modeling; a fusion architecture of multi-scale strip attention, frequency domain adaptive enhancement, and state-space models has not been reported. Fourth, existing methods exhibit poor robustness to interference factors such as occlusion, pose changes, and uneven illumination. They lack effective feature refinement mechanisms to accurately locate the most discriminative expression-related regions and generally suffer from model training bias and overfitting issues caused by class imbalance. Therefore, constructing a facial expression recognition method that can achieve synergistic enhancement of spatial and frequency domain features, efficient global dependency modeling, and accurate focusing of key regions remains a pressing technical challenge in this field. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a bidirectional Mamba facial expression recognition method based on deep fusion enhancement, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a bidirectional Mamba facial expression recognition method based on deep fusion enhancement, comprising: Step S1: Preprocess and augment the facial expression images in the training set, and preprocess the facial expression images in the test set; Step S2: Using ResNet18 as the backbone network, a parallel dual-branch structure is constructed after layer 3 of ResNet18. The first branch sequentially performs multi-scale strip-gated attention enhancement and complex frequency domain adaptive enhancement on the mid-level spatial features. The second branch performs bidirectional Mamba sequence long-range dependency modeling on the mid-level spatial features. The outputs of the first branch and the second branch are fused to obtain the fused enhanced features. After layer 4 of ResNet18, the fused enhanced features are sequentially processed by two-dimensional bidirectional Mamba aggregation and sparse attention refinement to obtain high-level semantic features. Step S3: Classify and predict the high-level semantic features, and use the focus loss with fused label smoothing as the loss function for model training; Step S4: Test the trained model using the test set.

[0006] Preferably, the multi-scale strip-gated attention enhancement in step S2 is achieved through a multi-scale strip-gated attention fusion module. The multi-scale strip-gated attention fusion module uses two dual-pooling strip-gated attention modules with different kernel scales for parallel fusion. Each dual-pooling strip-gated attention module consists of a horizontal strip-gated attention sub-module and a vertical strip-gated attention sub-module connected in series.

[0007] Preferably, the horizontal strip gated attention submodule is configured with a horizontal strip convolution of the first kernel size, and the vertical strip gated attention submodule is configured with a vertical strip convolution of the second kernel size; each submodule is provided with a depth-separable convolution unit, a point convolution unit and a dual pooling gate unit, and the dual pooling gate unit is configured with adaptive average pooling and adaptive max pooling in parallel, and the two outputs are multiplied element-wise with the output of the point convolution unit after channel splicing and activation.

[0008] Preferably, the complex frequency domain adaptive enhancement in step S2 is implemented by a complex frequency domain adaptive enhancement module. The complex frequency domain adaptive enhancement module performs a two-dimensional Fourier transform on the input features, and separates the frequency domain features into low-frequency components and high-frequency components based on learnable cutoff frequency parameters. After being processed by complex convolution units, the components are restored to the spatial domain by inverse Fourier transform, and then spatial domain enhancement is performed by real convolution units. The enhanced low-frequency response and high-frequency response are then concatenated and output with channel attention weighting.

[0009] Preferably, the bidirectional Mamba sequence long-range dependency modeling in step S2 is implemented through a bidirectional Mamba fusion module. The bidirectional Mamba fusion module receives sequence format input features, performs global dependency modeling through a multi-head self-attention unit, and then sends them in parallel to an independent forward Mamba unit and an independent reverse Mamba unit. The independent reverse Mamba unit performs reverse scanning by flipping the sequence. The two outputs are added and fused, and then subjected to nonlinear transformation by a multilayer perceptron unit.

[0010] Preferably, the two-dimensional spatial bidirectional Mamba aggregation processing in step S2 is implemented by an enhanced bidirectional Mamba aggregation module. The enhanced bidirectional Mamba aggregation module reshapes the features after multi-head self-attention into a two-dimensional spatial feature map, expands it into a sequence along the height and width directions respectively, and then performs a forward Mamba scan in sequence. After further restoring it into a two-dimensional spatial feature map and performing a dimension flip, it performs a reverse Mamba scan in sequence along the height and width directions. The scans in each direction are serially connected and output.

[0011] Preferably, the sparse attention refinement process in step S2 is implemented by a sparse attention refinement module, which includes a TopK sparse attention unit and a global context fusion feedforward unit. The TopK sparse attention unit retains the Top-K attention score after query-key-value projection and outputs it with weights. The global context fusion feedforward unit performs global context aggregation and normalization to enhance channel attention weights on the input features and then fuses the output.

[0012] Preferably, the global context fusion feedforward unit performs spatial pooling on the two-dimensional spatial feature map to obtain a global context vector, generates channel attention weights based on the weight parameters of the batch normalization layer, and performs spatial fusion enhancement. The enhanced channel attention features are then used to perform weighted modulation on the combination of the two-dimensional spatial feature map and the global context vector.

[0013] Preferably, in step S3, the focus loss with fused label smoothing performs label smoothing on the probability of each class predicted by the model, and assigns higher loss weights to hard-classified samples based on the focus parameter. The label smoothing is used to suppress overfitting, and the focus parameter is used to adjust the loss contribution ratio of easy and hard samples.

[0014] In a second aspect, the present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention constructs a first branch after layer 3 of ResNet18, where multi-scale strip gating attention enhancement can extract spatial attention features under different receptive fields along the horizontal and vertical directions, effectively enhancing the network's local perception ability of cross-regional facial texture and contour information; complex frequency domain adaptive enhancement deeply mines frequency domain details that are difficult to obtain directly in the spatial domain through adaptive separation of high and low frequency components and complex convolution processing. The two work together to enable the network to achieve complementary enhancement of local spatial features and global frequency information in the mid-level feature extraction stage, solving the limitation of conventional spatial domain convolution in fully mining frequency domain information.

[0016] This invention constructs a second branch after layer 3 of ResNet18 to perform bidirectional Mamba sequence long-range dependency modeling on mid-level spatial features. This captures long-distance contextual associations between different facial regions with linear computational complexity. The output of this branch is then fused with the output of the first branch, enabling the network to maintain its lightweight advantage while possessing global semantic awareness capabilities. This overcomes the inherent limitation of the receptive field in pure convolutional networks, and the computational cost is significantly lower than the quadratic complexity self-attention mechanism of the Transformer architecture.

[0017] This invention applies enhanced features to a two-dimensional bidirectional Mamba aggregation process followed by sparse attention refinement after layer 4 of ResNet18. The two-dimensional bidirectional Mamba aggregation performs forward and reverse scanning along both the height and width directions, achieving global dependency modeling in each direction of the two-dimensional space with linear complexity. Compared to conventional one-dimensional sequence Mamba scanning, it has a stronger modeling ability for the asymmetric distribution and coordinated changes of facial muscles in the two-dimensional space. The sparse attention refinement uses the TopK mechanism to select key attention connections and combines them with global context for channel reweighting, enabling the network to focus on the most discriminative expression regions. This effectively suppresses the influence of interference factors such as complex backgrounds, occlusion, and pose changes, improving the model's robustness in complex scenes.

[0018] This invention employs a focus loss with fused label smoothing as the loss function. The label smoothing mechanism softens the distribution of the true labels to prevent the network from becoming overconfident in the training set. At the same time, the focus parameter assigns higher loss weights to hard-to-classify samples to enhance the network's learning of difficult samples. The two work together to effectively improve the training stability and generalization performance of the model under imbalanced class conditions. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a bidirectional Mamba facial expression recognition method based on deep fusion enhancement according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the Dual-Pooling Stripe Gated Attention Module (DSGAM) according to an embodiment of the present invention. Figure 3This is a schematic diagram of the Complex Frequency-Domain Adaptive Enhancement Module (CFAEM) according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of the Bidirectional Mamba FusionModule (BMFM) according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of the Enhanced Bidirectional Mamba Aggregation Module (EBAM) according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the results of the Sparse Attention Refinement Module (SARM) in an embodiment of the present invention. Figure 7 This is a schematic diagram of the facial expression recognition confusion matrix of the RAF-DB and AffectNet datasets in an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0022] Example 1 This embodiment provides a bidirectional Mamba facial expression recognition method based on deep fusion enhancement, including: Step S1: Preprocess and augment the facial expression images in the training set, and preprocess the facial expression images in the test set; Specifically, this embodiment uses a publicly available facial expression dataset containing 7-8 expression labels (surprise, fear, disgust, happiness, sadness, anger, neutrality, and contempt). The expression dataset is divided into training and testing sets. The training input images are preprocessed and enhanced to improve model robustness and avoid overfitting. Specifically, this includes scaling the uniform facial expression images, randomizing them five times, randomly flipping them horizontally, normalizing them, and randomly erasing them. The testing input images are scaled, center-cropped, and normalized, without random enhancement, to ensure the objectivity of the validation results.

[0023] Step S2: Using ResNet18 as the backbone network, a parallel dual-branch structure is constructed after layer 3 of ResNet18. The first branch sequentially performs multi-scale strip-gated attention enhancement and complex frequency domain adaptive enhancement on the mid-level spatial features. The second branch performs bidirectional Mamba sequence long-range dependency modeling on the mid-level spatial features. The outputs of the first branch and the second branch are fused to obtain the fused enhanced features. After layer 4 of ResNet18, the fused enhanced features are sequentially processed by two-dimensional bidirectional Mamba aggregation and sparse attention refinement to obtain high-level semantic features. Specifically, this embodiment uses ResNet18 as the backbone network, loads pre-trained weights, and adjusts the output dimension of the fully connected layers to adapt to the requirements of facial expression classification. Layer 1 to layer 2 is the low-level feature extraction stage, capturing basic textures and other low-level features. Adding complex modules will increase computational overhead and easily lead to overfitting. Layer 3 to layer 4 is the mid-to-deep feature extraction stage, capturing local textures and global semantic features. Therefore, the network structures of layer 1 and layer 2 are not changed, and feature enhancement modules are only inserted after layer 3 and layer 4.

[0024] The specific approach is as follows: After layer 3 of the ResNet18 network, a dual-branch structure is introduced. Branch 1 sequentially enhances the joint features in the spatial and frequency domains through a multiscale stripe attention fusion module (MSAFM) cascaded with a complex frequency-domain adaptive enhancement module (CFAEM). Simultaneously, Branch 2 completes one-dimensional sequence long-range dependency modeling through a bidirectional Mamba fusion module (BMFM). It then performs long-range dependency modeling of the enhanced features through a cascaded bidirectional one-dimensional Mamba fusion structure using Transformer's multi-head self attention (MHSA), compensating for the insufficient long-range dependency modeling of mid-layer features. Finally, the dual-branch fusion collaboratively achieves the fusion enhancement of joint spatial-frequency domain feature extraction and one-dimensional long-range dependency modeling.

[0025] Furthermore, the multi-scale strip-gated attention enhancement in step S2 is achieved through a multi-scale strip-gated attention fusion module. The multi-scale strip-gated attention fusion module uses two dual-pooling strip-gated attention modules at different kernel scales for parallel fusion. Each dual-pooling strip-gated attention module consists of a horizontal strip-gated attention sub-module and a vertical strip-gated attention sub-module connected in series.

[0026] Furthermore, the horizontal strip gated attention submodule is configured with a horizontal strip convolution of the first kernel size, and the vertical strip gated attention submodule is configured with a vertical strip convolution of the second kernel size; each submodule is equipped with a depth-separable convolution unit, a point convolution unit, and a dual-pooling gated unit. The dual-pooling gated unit is configured with adaptive average pooling and adaptive max pooling in parallel. The two outputs are multiplied element-wise with the output of the point convolution unit after channel concatenation and activation.

[0027] Specifically, the Multi-Scale Stripe Attention Fusion Module (MSAFM) employs two Dual-Pooling Stripe Gated Attention Modules (DSGAM) at different kernel scales for parallel addition and fusion. Each scale's dual-pooling stripe gated attention consists of two cascaded sub-strip attention modules. The first sub-module is configured with a horizontal stripe convolution with a kernel size of (1,k), and the second sub-module is configured with a vertical stripe convolution with a kernel size of (k,1). Each sub-module contains depthwise separable convolutional units with the number of groups equal to the number of input channels, as well as 1×1 point convolutional units. Each sub-module also contains adaptive average pooling units and adaptive max pooling units in parallel. The two outputs are concatenated through channels and then gating weights are generated by 1×1 convolution and sigmoid activation, which are then multiplied element-wise with the output of the point convolutional unit.

[0028] In this embodiment, MSAFM is constructed by additive fusion of dual-pooling strip-gated attention modules (DSGAM) using different kernel sizes. Each kernel size (e.g., k=5 or k=3) employs two cascaded strip convolutional blocks. First, a horizontal strip convolution with a kernel size of (1,5) is used, followed by depthwise separable convolution to reduce computation. Cross-channel interaction via point convolution is then used to obtain the horizontal strip attention map. We continue to use a dual-pooling gating mechanism, generating attention weights through parallel adaptive average pooling and adaptive max pooling, followed by concatenation, convolution, and sigmoid. Multiplying this by the attention map above achieves soft attention weighting, calculated using the following formula: (1) (2) In the formula, Attention weights; Sigmoid is the activation function; This is a 1×1 convolution operation; For channel splicing; and These are adaptive average pooling and adaptive max pooling, respectively.

[0029] Following this, cascaded vertical stripe convolutions with a kernel size of (5,1) are used in the above pattern. Based on this, obtain vertical strip attention weighting Ultimately, this is achieved through learnable scaling parameters. γ and β Residual fusion is performed as shown in the following equation: (3) In the formula, For input features; This is the final output feature of the module; These are the attention weights after sequential horizontal and vertical weighting; and For learnable scaling parameters.

[0030] Furthermore, the complex frequency domain adaptive enhancement described in step S2 is implemented through a complex frequency domain adaptive enhancement module. The complex frequency domain adaptive enhancement module performs a two-dimensional Fourier transform on the input features, and separates the frequency domain features into low-frequency components and high-frequency components based on learnable cutoff frequency parameters. After being processed by complex convolution units, the components are restored to the spatial domain by inverse Fourier transform, and then spatial domain enhancement is performed by real convolution units. The enhanced low-frequency response and high-frequency response are then concatenated and convolved, and finally weighted with the attention of the SE channel for output.

[0031] Specifically, the Complex Frequency Domain Adaptive Enhancement Module (CFAEM) performs a two-dimensional fast Fourier transform on the input features, generates high and low frequency masks using learnable cutoff frequency parameters and spatial distance grids, and separates the low and high frequency components. The low and high frequency components are processed by complex convolution units, and after inverse Fourier transform to extract the real parts and convert them back to the spatial domain, they are further enhanced by real convolution units to extract high and low frequency features respectively. The high and low frequency response features are concatenated by channels and then weighted by channel attention gating units to obtain the final output.

[0032] In this embodiment, the output of the MSAFM is received. As input, a two-dimensional fast Fourier transform (2D FFT) is performed on the input features to generate a frequency domain feature map, as shown in the following equation: (4) In the formula, This is a two-dimensional Fast Fourier Transform. The spectrum is then shifted to the central region to obtain the complex frequency domain characteristics. .

[0033] Next, a high-low frequency separation mask is generated using a learnable cutoff frequency parameter, as shown in the following equation: (5) (6) In the formula, It is a learnable cutoff frequency, which can be automatically learned through network training without manual setting; This is the normalized Euclidean distance from each point on the feature map to the center point; Use the Sigmoid activation function; and These are low-frequency masks and high-frequency masks, respectively; respectively with and Multiplying yields the low-frequency component in the frequency domain. and high frequency components As shown in the following formula: (7) (8) After separating the frequency domain features into low-frequency and high-frequency components, feature extraction is performed separately using complex convolution to obtain the enhanced frequency domain representations, which are respectively... and Complex convolution can simultaneously learn the amplitude and phase information of the spectrum, thus improving frequency domain modeling capabilities.

[0034] The enhanced frequency domain features are restored to the spatial domain by inverse Fourier transform (IFFT), yielding low-frequency and high-frequency spatial features respectively.

[0035] To further integrate local spatial context information, the restored low-frequency and high-frequency spatial features are processed through independent real-number convolutional blocks (containing convolution, batch normalization, and GELU activation function) to ensure the complementarity between global information in the frequency domain and local features in the spatial domain.

[0036] The low-frequency and high-frequency spatial features enhanced by real-valued convolutions are filtered separately. For low-frequency features, average pooling (AvgPool2d) followed by a 1×1 convolution is used to extract smooth contextual information. For high-frequency features, depthwise convolution followed by a 1×1 convolution is used to enhance the edge texture details of the high-frequency features. Then, channel concatenation and fusion are performed.

[0037] To adaptively calibrate the channel importance of fused features, this invention introduces an SE channel attention gating mechanism. First, the original input features are compressed globally using spatial information, and the nonlinear dependencies between channels are learned through a multilayer perceptron (composed of 1×1 convolutions and activation functions). Finally, an adaptive gating weight vector with values ​​ranging from 0 to 1 is generated via a sigmoid function. Then, the feature vector, after being processed and fused in both the frequency and spatial domains, is dimensionally projected through a 1×1 convolution and element-wise multiplied with the adaptive gating weight vector. This operation enhances key feature channels and suppresses redundant noise channels, thus outputting the final feature enhancement result.

[0038] Furthermore, the bidirectional Mamba sequence long-range dependency modeling in step S2 is achieved through a bidirectional Mamba fusion module. The bidirectional Mamba fusion module receives sequence format input features, performs global dependency modeling through a multi-head self-attention unit, and then sends them in parallel to an independent forward Mamba unit and an independent reverse Mamba unit. The independent reverse Mamba unit performs reverse scanning by flipping the sequence. The two outputs are added and fused, and then subjected to nonlinear transformation by a multilayer perceptron unit.

[0039] Specifically, the bidirectional Mamba fusion module (BMFM) receives sequence-formatted input features, which are then normalized and fed into a multi-head self-attention unit for global dependency modeling. The output is then connected to the input residual. After normalization, the features are fed into independent forward Mamba units and independent reverse Mamba units in parallel. The reverse unit performs a reverse scan by flipping the sequence. The two outputs are added together and connected to the input residual. Finally, the features are fed into a multilayer perceptron unit after normalization, and the residuals are connected to the output to enhance the features.

[0040] In this embodiment, the bidirectional Mamba fusion module (BMFM) includes a layer normalization unit, a multi-head self-attention unit, a bidirectional Mamba unit, a multilayer perceptron unit, and a residual connection unit. The layer normalization unit is respectively located at the input ends of the multi-head self-attention unit, the bidirectional Mamba unit, and the multilayer perceptron unit. The multi-head self-attention unit is configured to perform query-key-value projection and multi-head attention weighting on the input sequence. The bidirectional Mamba unit is cascaded with the multi-head self-attention unit and configured to simultaneously perform independent forward Mamba scanning and independent reverse Mamba scanning on the attention output. The reverse scanning is achieved by sequence flipping, and the two outputs are added and fused. The multilayer perceptron unit is cascaded with the bidirectional Mamba unit and configured to perform channel-dimensional nonlinear transformation on the fused features. The units are added and output through residual connections.

[0041] The Mamba parameters are configured as follows: (9) (10) (11) In the formula, Hidden state for continuous time The first derivative with respect to time; , For a continuous system, the input and output sequences are given. , , The first k Discrete hidden states, inputs, and outputs of each step; , , , It is a continuous state-space parameter matrix; , , , This corresponds to the discretized parameter matrix; This is the discretization step size; It is the identity matrix; It is a time variable; This is for matrix exponentiation operations.

[0042] Furthermore, the two-dimensional spatial bidirectional Mamba aggregation processing described in step S2 is implemented by an enhanced bidirectional Mamba aggregation module. The enhanced bidirectional Mamba aggregation module reshapes the features after multi-head self-attention into a two-dimensional spatial feature map, unfolds it into a sequence along the height and width directions respectively, and then performs a forward Mamba scan. Then, after restoring it to a two-dimensional spatial feature map, it performs a dimension flip and then performs a reverse Mamba scan along the height and width directions respectively. The scan outputs in each direction are connected serially.

[0043] Furthermore, the sparse attention refinement process in step S2 is implemented by a sparse attention refinement module, which includes a TopK sparse attention unit and a global context fusion feedforward unit. The TopK sparse attention unit retains the Top-K attention score after query-key-value projection and outputs it with weights. The global context fusion feedforward unit performs global context aggregation and normalization to enhance channel attention weights on the input features and then fuses the output.

[0044] Furthermore, the global context fusion feedforward unit performs spatial pooling on the two-dimensional spatial feature map to obtain a global context vector, generates channel attention weights based on the weight parameters of the batch normalization layer and performs spatial convolution enhancement, and uses the enhanced channel attention weights to perform weighted fusion modulation with the global context vector.

[0045] Specifically, after layer 4 of the ResNet18 network, the output features are fed into the Enhanced Bidirectional Mamba Aggregation Module (EBMAM) to complete deeper spatial global dependency modeling. Then, the Sparse Attention Refinement Module (SARM) completes the key region feature selection and focusing to obtain high-level semantic features. The two are connected in series. EBMAM receives spatial features and performs sequential Mamba bidirectional scanning along the height and width directions in the forward direction based on multi-head attention. It continues to perform sequential Mamba bidirectional scanning along the height and width directions in the reverse direction by sequence flipping to achieve two-dimensional spatial dependency modeling. Finally, it completes global dependency modeling in each direction with linear complexity. Its output features are input into SARM for key feature region focusing. The fusion of the above two modules realizes the global optimization and accurate characterization of deep facial expression features.

[0046] In this embodiment, the layer 4 feature enhancement stage serves as the core of deep feature extraction, focusing on achieving global spatial dependency modeling of two-dimensional sequences and key region feature focusing. It constructs two enhancement modules: an enhanced bidirectional Mamba aggregation module (EBMAM) and a sparse attention refinement module (SARM). These two modules are connected in series. First, EBMAF performs forward and backward bidirectional Mamba scans along the height and width directions to complete global spatial dependency modeling. Then, its output features are input into SARM to focus on key facial expression regions and suppress redundant interference. Finally, the two modules work together to output high-level semantic features.

[0047] The core modules specifically include: 1) Enhanced Bidirectional Mamba Aggregation Module (EBMAM): Includes a multi-head self-attention unit, a spatial scanning unit, and a multilayer perceptron unit; the multi-head self-attention unit is configured with query-key-value projection and multi-head attention weighting; the Mamba spatial scanning unit is configured to reshape features into a two-dimensional grid, unfold the sequence sequentially along the height and width directions, and perform forward and reverse Mamba scans, with the reverse scan achieved by sequence flipping, and each direction scan is executed in a concatenated mode; the multilayer perceptron unit is configured with channel dimension nonlinear transformation; the units are connected through residual connections and layer normalization connections.

[0048] The processing flow of the enhanced bidirectional Mamba aggregation module includes: First, the input serialized facial feature tensor undergoes a first-layer normalization. After extracting global long-range dependency features through a multi-head self-attention mechanism, it is fused with the original input features via residual connections to obtain a first intermediate feature. Second, the first intermediate feature undergoes a second-layer normalization and is reshaped into a two-dimensional spatial feature map, which is then sequentially entered into bidirectional two-dimensional state space modeling. Specifically, the forward scanning path utilizes the first two-dimensional sequence Mamba module to sequentially perform feature scanning and state space transformation along the horizontal and vertical dimensions to capture forward spatial context information; the reverse scanning... The path involves dimensional flipping the two-dimensional feature map, using the second two-dimensional sequence Mamba module to scan features and transform and restore the state space along the flipped horizontal and vertical dimensions to capture the reverse space context information. The context information that enters the forward and reverse space outputs sequentially is flattened into serialized features, and then added to the first intermediate feature through residual connection to obtain the second intermediate feature. Finally, the second intermediate feature is normalized in the third layer, and nonlinear feature mapping and channel interaction are performed through multilayer perceptron branches. Its output is again fused with the second intermediate feature through residual connection to output the final enhanced facial expression features.

[0049] 2) Sparse Attention Refinement Module (SARM): This module includes a Top-K Sparse Attention Unit, a Global Context Feedforward Unit, and a Residual Connection Unit. The Top-K Sparse Attention Unit is used for global dependency modeling and feature selection of the input feature sequence. This module queries the input features, maps keys to values, calculates the global attention similarity matrix, and filters attention connections based on a preset or adaptive Top-K strategy, retaining only the key response relationships at each query position, while subjecting the remaining relationships to sparsification constraints. Subsequently, the sparsified attention matrix is ​​normalized, and an output feature representation is generated based on a weighted summation method. This module effectively reduces redundant computation while ensuring global information modeling capabilities, improving the model's ability to model key feature relationships and computational efficiency.

[0050] The global context feedforward unit is cascaded with the Top-K sparse attention unit. By performing channel normalization and learnable channel weight modeling on the input features, it enhances channel-level attention allocation. Simultaneously, a spatial context pooling mechanism is introduced to perform global semantic aggregation on the feature maps. Furthermore, the enhanced channel attention is fused with global context features to achieve adaptive reweighting of feature responses. Channel attention supplements and enhances spatial structure information through deep convolution, thereby simultaneously improving the global semantic representation and local spatial modeling capabilities of the features. This module is suitable for visual feature enhancement and discriminative learning tasks in complex scenarios.

[0051] The residual connection units are respectively set at the outputs of the Top-K sparse attention unit and the global context feedforward unit; the layer normalization units are respectively set at the inputs of each unit.

[0052] The input feature map is reshaped into a sequence format and then fed into the TopK sparse attention unit. After query-key-value projection, the Top-K attention is retained, and the result is normalized weighted sum and output through the first residual connection unit. The global context feedforward unit includes a global context extraction unit, a normalized enhanced channel attention unit, and a feature fusion unit. The global context extraction unit is used to perform spatial pooling on the two-dimensional spatial feature map to generate a normalized global context vector. The normalized enhanced channel attention unit receives the two-dimensional spatial feature map, generates channel attention weights based on batch normalization layer parameters, and simultaneously introduces deep convolution to extract spatial features to achieve spatial enhancement. The weighted channel features and spatial enhancement features are fused to obtain the output features. The feature fusion unit is used to use the attention features to perform weighted modulation on the combination of the two-dimensional spatial feature map and the global context vector, and the resulting fused spatial feature map is output through the second residual connection unit.

[0053] Step S3: Classify and predict the high-level semantic features, and use the focus loss with fused label smoothing as the loss function for model training; Furthermore, in step S3, the focus loss with fused label smoothing performs label smoothing on the probability of each class predicted by the model, and assigns higher loss weights to hard-classified samples based on the focus parameter. The label smoothing is used to suppress overfitting, and the focus parameter is used to adjust the loss contribution ratio of easy and hard samples.

[0054] Specifically, global average pooling is performed on high-level semantic features, and the expression category probability is output through a fully connected layer; focal loss with fused label smoothing is used as the loss function, and the model is trained using an SGD optimizer and a step-wise learning rate scheduler.

[0055] This embodiment combines label smoothing and focus loss as loss functions for supervised learning. For hard-to-classify samples, the loss weights are dynamically increased, while label softening prevents overfitting. The calculation formula is as follows: (12) (13) In the formula, For the first label smoothing Class target probability; The true labels for the samples; The model predicts the first k Class probability; For the first k Class weighting factor; For focusing parameters; For label smoothing factor; K Total number of categories; For the Kronecker delta function (when k = y (If it is 1, then it is 0; otherwise, it is 0). This is the total loss function that combines label smoothing and focus loss.

[0056] Step S4: Test the trained model using the test set.

[0057] The beneficial effects of this embodiment: This embodiment designs a deeply fused and enhanced bidirectional Mamba method for facial expression recognition in complex natural scenes. First, features are extracted in the spatial-frequency domain and fused with bidirectional Mamba joint modeling to effectively enhance the spatial perception and local fine-grained perception of expressions. Furthermore, an enhanced bidirectional Mamba aggregation module is used to achieve more effective two-dimensional sequence global modeling with a forward and reverse bidirectional scanning mode along the height and width multidimensional directions and linear complexity. This can more fully model the dependencies between various regions of the face and capture subtle and asymmetrical expression features. Then, a sparse attention refinement module is used to better focus on the most discriminative features, enabling the model to achieve a significantly higher recognition rate and robustness than existing methods.

[0058] This embodiment uses a stage feature enhancement based on the ResNet18 baseline model. The structure is simple and has a small number of parameters and low complexity, making it suitable for image classification. The focus loss based on label smoothing can further supervise network learning to achieve focusing on hard samples and modeling label uncertainty, thus making the model more generalizable.

[0059] Example 2 Based on the technical solution of Embodiment 1, this embodiment provides a bidirectional Mamba facial expression recognition method based on deep fusion enhancement, such as... Figure 1 As shown, it includes the following steps: Step S1: Data Preprocessing and Augmentation. The detected natural facial expressions are divided into training and test sets. The training set is subjected to scaling, random cropping, random flipping, random erasing, and normalization. The test set is subjected to scaling, center cropping, and normalization.

[0060] Step S2: Construct a multi-stage, deeply fused, enhanced bidirectional Mamba feature extraction network model. For example... Figure 2As shown, the Layer 3 output (256 channels, output size 256×14×14) is first fed into dual-pooling strip-gated attention modules (DSGAM) with different kernel scales (k=5) and (k=3) for addition and fusion. It is then further processed by a complex frequency domain adaptive enhancement module (CFAEM). Simultaneously, the Layer 3 output is flattened into a sequence and input into a bidirectional Mamba fusion module (BMFM) for contextual feature enhancement to restore the image spatial structure. Finally, the two branch structures are fused and added in parallel to restore the (256, 14, 14) size.

[0061] The Layer 4 output (512 channels, 512×7×7) is flattened into a sequence and input into the Enhanced Bidirectional Mamba Aggregation Module (EBMAF) for multi-directional two-dimensional sequence modeling. The result is then input into the Sparse Attention Aggregation Module (SARM) to select features using Top-K sparse attention. Furthermore, the spatially enhanced channel attention features are interactively fused with global context features to obtain enhanced feature representations, improving the discriminativeness and robustness of feature representations in complex scenes. The output undergoes global average pooling and is connected to a fully connected layer for final classification.

[0062] The key parameter configurations for each module are as follows: like Figure 3 As shown, the complex convolution in the CFAEM module uses grouped convolution, the real convolution uses a 3×3 kernel, the channel dimensionality reduction ratio is 16, and the cutoff frequency is initialized to 0.5 and is a trainable parameter.

[0063] like Figure 4 As shown, the BMAF module has 8 multi-head attention (MSHA) heads, the Mamba module sets the state space dimension to 64 to control the memory capacity of the sequence state representation, sets the local convolution kernel size to 4 to enhance the modeling ability of local sequence features, and sets the channel expansion factor to 4 to improve the non-linear expressive ability of feature representation and the feature mapping ability of the model.

[0064] like Figure 5 As shown, the EBMAF module performs serial forward and reverse scanning. Forward scanning instances perform scanning sequentially along the height and width directions, while reverse scanning instances perform reverse scanning sequentially along the height and width directions by feature flipping, ultimately achieving traversal scanning of the two-dimensional spatial sequence.

[0065] like Figure 6As shown, the Top-K sparse attention module in the SARM module sets the number of multi-head attention to 8 to achieve feature subspace partitioning; the feature dimension is evenly distributed according to the number of heads to enhance expressive power; the Top-K parameter is set to 64 to retain only 64 most relevant attention connections for each query position, and the remaining connections are sparsified to reduce computational redundancy and improve the ability to model key features. In addition, the stacking depth of the SARM module is 3.

[0066] The model was trained on a publicly available facial expression dataset and optimized using the SGD+Momentum optimizer with a batch size of 32. The learning rate was decayed by a factor of 10 every 20 iterations on the RAF-DB dataset for 60 training epochs, and by a factor of 10 every 5 iterations on the AffectNet dataset for 10 training epochs.

[0067] Step S3: Classification and loss calculation. A joint supervised training strategy is adopted, which combines label smoothing loss with a smoothing coefficient of 0.1 with focal loss.

[0068] Step S4: Test the facial expression recognition model using the test set.

[0069] like Figure 7 As shown, the model was tested on the RAF-DB and AffectNet large-scale natural facial expression datasets, achieving accuracies of 97.33% and 74.80% respectively, which are significant improvements over the baseline network and the current state-of-the-art method. This demonstrates that the bidirectional Mamba facial expression recognition method with segmented deep fusion enhancement is very effective.

[0070] Example 3 This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0071] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A bidirectional Mamba facial expression recognition method based on deep fusion enhancement, characterized in that, Includes the following steps: Step S1: Preprocess and augment the facial expression images in the training set, and preprocess the facial expression images in the test set; Step S2: Using ResNet18 as the backbone network, a parallel dual-branch structure is constructed after layer 3 of ResNet18. The first branch performs multi-scale strip gating attention enhancement and complex frequency domain adaptive enhancement on the mid-level spatial features in sequence. The second branch performs bidirectional Mamba sequence long-range dependency modeling on the mid-level spatial features. The outputs of the first branch and the second branch are fused to obtain the fused enhanced features. After layer 4 of the ResNet18, the fused and enhanced features are sequentially processed by two-dimensional bidirectional Mamba aggregation and sparse attention refinement to obtain high-level semantic features. Step S3: Classify and predict the high-level semantic features, and use the focus loss with fused label smoothing as the loss function for model training; Step S4: Test the trained model using the test set.

2. The method according to claim 1, characterized in that, The multi-scale strip-gated attention enhancement in step S2 is achieved through a multi-scale strip-gated attention fusion module. The multi-scale strip-gated attention fusion module uses two dual-pooling strip-gated attention modules with different kernel scales for parallel fusion. Each dual-pooling strip-gated attention module consists of a horizontal strip-gated attention sub-module and a vertical strip-gated attention sub-module connected in series.

3. The method according to claim 2, characterized in that, The horizontal strip gated attention submodule is configured with a horizontal strip convolution with a first kernel size, and the vertical strip gated attention submodule is configured with a vertical strip convolution with a second kernel size. Each submodule is equipped with a depth-separable convolution unit, a point convolution unit, and a dual-pooling gated unit. The dual-pooling gated unit is configured with adaptive average pooling and adaptive max pooling in parallel. The two outputs are concatenated and activated by channels and then multiplied element-wise with the output of the point convolution unit.

4. The method according to claim 1, characterized in that, The complex frequency domain adaptive enhancement described in step S2 is implemented by a complex frequency domain adaptive enhancement module. The complex frequency domain adaptive enhancement module performs a two-dimensional Fourier transform on the input features and separates the frequency domain features into low-frequency and high-frequency components based on the learnable cutoff frequency parameter. After being processed by complex convolution units, the components are restored to the spatial domain by inverse Fourier transform and then enhanced in the spatial domain by real convolution units. The enhanced low-frequency and high-frequency responses are then concatenated and output with channel attention weighting.

5. The method according to claim 1, characterized in that, The bidirectional Mamba sequence long-range dependency modeling in step S2 is achieved through a bidirectional Mamba fusion module. The bidirectional Mamba fusion module receives sequence format input features, performs global dependency modeling through a multi-head self-attention unit, and then sends them in parallel to an independent forward Mamba unit and an independent reverse Mamba unit. The independent reverse Mamba unit performs reverse scanning by flipping the sequence. The two outputs are added and fused, and then subjected to nonlinear transformation by a multilayer perceptron unit.

6. The method according to claim 1, characterized in that, The two-dimensional spatial bidirectional Mamba aggregation processing described in step S2 is implemented by an enhanced bidirectional Mamba aggregation module. The enhanced bidirectional Mamba aggregation module reshapes the features after multi-head self-attention into a two-dimensional spatial feature map, expands it into a sequence along the height and width directions respectively, and then performs a forward Mamba scan in sequence. After recovering the two-dimensional spatial feature map, it performs a dimension flip and performs a reverse Mamba scan in sequence along the height and width directions. The scan results in each direction are output serially.

7. The method according to claim 1, characterized in that, The sparse attention refinement process described in step S2 is implemented through a sparse attention refinement module, which includes a TopK sparse attention unit and a global context fusion feedforward unit. The TopK sparse attention unit retains the Top-K attention score after query-key-value projection and outputs it with weights. The global context fusion feedforward unit performs global context aggregation and normalization to enhance channel attention weights on the input features and then fuses the output.

8. The method according to claim 7, characterized in that, The global context fusion feedforward unit performs spatial pooling on the two-dimensional spatial feature map to obtain a global context vector, generates channel attention based on the weight parameters of the batch normalization layer and performs spatial convolution enhancement, and uses the enhanced channel attention features to perform weighted modulation on the combination of the two-dimensional spatial feature map and the global context vector.

9. The method according to claim 1, characterized in that, In step S3, the focus loss with fused label smoothing performs label smoothing on the probability of each class predicted by the model, and assigns higher loss weights to hard-classified samples based on the focus parameter. The label smoothing is used to suppress overfitting, and the focus parameter is used to adjust the loss contribution ratio of easy and hard samples.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-9.

Images