A multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion

Abstract

This invention relates to the field of facial expression recognition technology, and in particular provides a multi-branch pyramid expression recognition method based on hierarchical spatial-frequency fusion. The method includes extracting pyramid features; employing a global semantic modeling module in the global branch, which guides a global attention mechanism to focus on expression-related regions by introducing facial landmark priors to model long-range dependencies; utilizing a local adaptive deformation perception module in the local branch, which captures texture changes in facial expressions through a kernel instantiation mechanism; employing a hierarchical spatial-frequency collaborative fusion module to reconstruct discriminative local-global details in the frequency domain using wavelet transform to obtain fused features; projecting the fused features into a set of feature vectors and inputting them into a lightweight VIT classifier to generate the final prediction result. This method, by constructing a multi-scale pyramid architecture, achieves deep collaboration between spatial semantics and frequency domain features, solving the problem of difficulty in capturing subtle expressions in facial expression recognition.

Description

Technical Field

[0001] This invention relates to the field of facial expression recognition technology, and in particular to a multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion. Background Technology

[0002] Facial Expression Recognition (FER) is a core technology in intelligent human-computer interaction, aiming to analyze and interpret human intentions and emotional states. It is widely used in various practical scenarios, including intelligent diagnosis and treatment, fatigue driving detection, and human-computer interaction systems. Despite significant progress, accurately capturing emotional states remains challenging, especially for low-intensity, subtle expressions. These expressions are characterized by subtle facial muscle movements, exhibiting weak discriminative features in the spatial domain, often making them difficult to distinguish from background noise or identity-related changes. Recent advanced methods, such as UA-FER, address this problem by introducing uncertainty-aware representation learning to mitigate the impact of ambiguous samples. Furthermore, the recognition of these expressions relies not only on local fine-grained deformations (such as slight movements around the eyes or mouth) but also on consistent interpretation based on global semantic context. This characteristic, involving local sparsity, global dependence, and high inter-class similarity, poses a significant challenge to traditional single-stream feature learning paradigms.

[0003] To improve representation capabilities, multi-branch fusion architectures have become an important direction in facial expression recognition (FER) research. By integrating complementary information from different branches, such as Transformer-based global dependency modeling, CNN-based local detail extraction, and landmark-guided attention mechanisms, these methods have demonstrated significant performance improvements. However, many existing fusion strategies primarily focus on the spatial domain. From the perspective of emotional visual analysis, different frequency components may encode complementary expression cues: low-frequency components are more related to global structural patterns, while high-frequency components are more sensitive to fine-grained textures, edges, and subtle changes caused by muscles. For subtle expression recognition, these frequency-aware cues are particularly important because they help decouple weak discriminative patterns from irrelevant appearance changes. Nevertheless, current multi-branch FER frameworks still lack a systematic mechanism to jointly model spatial semantics, multi-band frequency characteristics, and their cross-domain interactions, which limits their ability to capture subtle facial changes for high-precision recognition. Summary of the Invention

[0004] In view of this, the present invention provides a multi-branch pyramid expression recognition method based on hierarchical spatial-frequency fusion to solve the problem of difficulty in capturing subtle expressions in facial expression recognition.

[0005] In a first aspect, the present invention provides a multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion, the method comprising:

[0006] Step 1: Given an input image of a human facial expression. Extracting pyramid features , Let C represent the feature at the i-th scale, and let C, H, and W represent the channel, image height, and width, respectively. Step 2: Based on Step 1, the global semantic modeling module L-GSM is used in the global branch. By introducing facial landmark priors, the global attention mechanism is guided to focus on expression-related regions in order to model remote dependencies. Step 3: Based on Step 1, in the local branch, the Local Adaptive Deformation Perception Module (LADP) is used to capture texture changes with direction perception capabilities in facial expressions through the kernel instantiation mechanism. Step 4: Based on Step 2 and Step 3, the hierarchical spatial-frequency collaborative fusion module HSFC is used to reconstruct discriminative local-global details in the frequency domain through wavelet transform, and adaptive dual-path fusion between spatial representation and frequency representation is performed to obtain fused features; Step 5: Based on step 4, project the fused features into a set of feature vectors and input them into a lightweight VIT classifier to generate the final prediction result.

[0007] Optionally, step 2 includes: First, a pre-trained and frozen landmark feature extractor is used to generate landmark-aware semantic features from the input facial expression image. Landmark-aware semantic features are used to encode the spatial distribution of expression-related key regions and serve as structural guidance information for global branch modeling. In each level of the pyramid, landmark-aware semantic features are linearly projected to form query vectors, while the corresponding backbone network features are projected to generate key and value vectors. Driven by queries guided by landmark information, global contextual information is integrated through aggregation based on a global attention mechanism. To enhance spatial awareness, a learnable relative position bias is incorporated into the global attention calculation process, the expression of which is: ; in, B represents the feature dimension and B represents the learnable relative position bias, which is used to supplement spatial position perception ability. Global attention output The final global features are obtained by refining them using a feedforward network. .

[0008] Optionally, step 3 includes: First, the pyramid features input at stage i are divided into two subspaces along the channel dimension: ,in These are used to model local spatial dependencies in the vertical and horizontal directions, respectively; for a specific spatial scale... The LADP module generates functions through a lightweight kernel. From shared kernel parameters A set of instance-aware convolutional kernels is derived, and their expressions are as follows: ; in, Represents a kernel generating function based on bilinear interpolation; Secondly, based on the instantiated directional convolution kernels, the LADP module performs a two-stage directional decomposition modeling. In the first stage, vertical and horizontal convolution kernels are applied to their respective subspaces to capture deformation cues along a single dimension, expressed as follows: ; ; in, Indicates the number of groups Depth-separable convolution, This represents the GELU activation function; In the second stage, cross-directional thinning is performed by exchanging the directional convolution kernels between the two subspaces, as expressed by: ; ; Finally, the LADP module outputs enhanced local features through residual connections. Its expression is: ; in, Maintaining the same spatial resolution as the input, the residual connections preserve the original orientation features. This indicates the splicing of channel dimensions.

[0009] Optionally, step 4 includes: The HSFC module adopts a dual-path adaptive learning framework, which aims to integrate spatial semantic feature representation with frequency domain texture cues. The design motivation of the HSFC module comes from prior knowledge: facial expression features have structural sparsity in the frequency domain, where low-frequency components correspond to global semantics and high-frequency components encode local deformations. By combining spectral decomposition, adaptive component selection and cross-domain collaboration, end-to-end spatial-frequency feature modeling is achieved. Step 41, Airspace Path; The spatial fusion path focuses on the efficient aggregation of multi-source features within the spatial domain. At the i-th pyramid level, local features... Global features And the fusion features propagated from the previous level The expression for concatenating along the channel dimension is: ; in, LN indicates concatenation along the channel dimension, and LN indicates layer normalization. Represents the GELU activation function; branch-specific convolutional layers , and The fused features, local features, and global features propagated from the previous level are projected into a unified embedding space. and Perform channel compression and remapping to ensure spatial blending; Step 42, Frequency Domain Path; The frequency domain path is used to decouple and enhance discriminative features from a spectral perspective. The HSFC module introduces Discrete Wavelet Transform (DWT) for frequency domain modeling in the parallel fusion path, and its expression is as follows: ; ; Where LL represents the low-frequency subband, and LH, HL and HH correspond to the high-frequency subbands along different directions, which together constitute the initial spectrum dictionary of facial expressions; Learnable parameters are then introduced. The expression for adaptive fusion across frequency bands is: ; To enhance the discriminative representation capability of frequency domain subbands, a cross-band attention mechanism is introduced to capture the potential correlations and complementary information between different frequency components. In the spectral distribution, the low-frequency subband contains the image's topological structure and global semantics. A channel attention mechanism is added to the fused low-frequency features, with the expression: ; in, This indicates element-wise multiplication. This represents the Sigmoid activation function. Two layers for channel excitation Convolutional layer composition; The high-frequency subband is used to capture edge and detail information of the image. Spatial attention guidance is added to the high-frequency path, and its expression is: ; Subsequently, the refined multi-band signal is reconstructed using the inverse discrete wavelet transform (IDWT) to generate a frequency-domain enhanced feature representation. ; Step 43: Residual Gating Refinement Module RGRB; The residual gating refinement module RGRB is designed. First, the adaptive dynamic selection of features is realized through the gating mechanism. Then, it is connected to an inverse residual MLP to build a complete inference unit from feature selection to deep reconstruction. To ensure effective integration of spatial semantic consistency and frequency domain accuracy, the HSFC module employs the RGRB module to balance the spatial and frequency domain paths, with the following expression: ; ; Finally, the HSFC module utilizes the inverse residual MLP to perform nonlinear manifold mapping in a 4-fold expanded high-dimensional space, the expression of which is: ; Where BN represents batch normalization used to accelerate convergence. This indicates a fusion feature that has been propagated from the previous level.

[0010] In a second aspect, embodiments of the present invention provide a computer-readable storage medium comprising a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to execute the multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion in the first aspect or any possible implementation thereof.

[0011] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the hierarchical spatial-frequency fusion-based multi-branch pyramid facial expression recognition method in the first aspect or any possible implementation of the first aspect.

[0012] The technical solution provided by this invention includes the following steps: given an input facial expression image, extract pyramid features; in the global branch, a global semantic modeling module (L-GSM) is used to guide a global attention mechanism to focus on expression-related regions by introducing facial landmark priors, in order to model long-range dependencies; in the local branch, a local adaptive deformation perception module (LADP) is used to capture texture changes with direction perception capabilities in facial expressions through a kernel instantiation mechanism; a hierarchical spatial-frequency collaborative fusion module (HSFC) is used to reconstruct discriminative local-global details in the frequency domain through wavelet transform, and adaptive dual-path fusion between spatial representation and frequency representation is performed to obtain fused features; the fused features are projected into a set of feature vectors and input into a lightweight VIT classifier to generate the final prediction result. This method, by constructing a multi-scale pyramid architecture, achieves deep collaboration between spatial semantics and frequency domain features, solving the problem of difficulty in capturing subtle expressions in facial expression recognition. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of a multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion provided in an embodiment of the present invention; Figure 2 A schematic diagram of the LADP module provided in an embodiment of the present invention; Figure 3 A schematic diagram of the HSFC module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0017] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0018] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0019] This invention provides a multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion, such as... Figures 1 to 3 As shown, the method includes: Step 1: Given an input image of a human facial expression. Extracting pyramid features , Let C represent the feature at the i-th scale, and let H and W represent the channel, image height, and width, respectively.

[0020] Step 2: Based on Step 1, the global semantic modeling module L-GSM is used in the global branch. By introducing facial landmark priors, the global attention mechanism is guided to focus on expression-related regions in order to model remote dependencies.

[0021] In this embodiment of the invention, step 2 includes: To capture long-range dependencies and model the overall facial structure, a Landmark-guided Global Semantic Modeling (L-GSM) module is introduced. This module utilizes facial landmark priors to guide the global attention mechanism. Figure 1 As shown, a pre-trained and frozen landmark feature extractor is first used to generate landmark-aware semantic features from the input facial expression image. Landmark-aware semantic features are used to encode the spatial distribution of expression-related key regions and serve as structural guidance information for global branch modeling. In each level of the pyramid, landmark-aware semantic features are linearly projected to form query (Q) vectors, while the corresponding backbone network features are projected to generate key (K) and value (V) vectors. Driven by the query guided by landmark information, global contextual information is integrated through aggregation based on a global attention mechanism. During this process, facial regions highly correlated with facial expression changes (such as eyebrows and mouth) are enhanced, while irrelevant background regions are suppressed. To enhance spatial awareness, a learnable relative position bias is incorporated into the global attention calculation process, expressed as: ; in, B represents the feature dimension and B represents the learnable relative position bias, which is used to supplement spatial position perception ability. Global attention output The final global features are obtained by refining them using a feed-forward network. Global features primarily encode overall facial structure information and long-distance semantic dependencies, providing stable global constraints for subsequent feature fusion stages.

[0022] Step 3: Based on Step 1, in the local branch, the Local Adaptive Deformation Perception Module (LADP) is used to capture texture changes with direction perception capabilities in facial expressions through the kernel instantiation mechanism.

[0023] To effectively capture these subtle local features, this invention introduces a Local Adaptive Deformation Perception Module (LADP) at each layer of the feature pyramid. Unlike standard convolutions that use fixed geometric kernels, the LADP module employs a meta-kernelinstantiation mechanism, which adaptively generates convolution weights for specific instances based on the spatial resolution of the current feature map. Subsequently, this module performs two-stage orientation decomposition modeling to flexibly model complex local texture structures.

[0024] In embodiments of the present invention, such as Figure 2 As shown, step 3 includes: First, the pyramid features input at stage i are divided into two subspaces along the channel dimension: ,in These are used to model local spatial dependencies in the vertical and horizontal directions, respectively; for a specific spatial scale... The LADP module generates functions through a lightweight kernel. From shared kernel parameters A set of instance-aware convolutional kernels is derived, and their expressions are as follows: ; in, This represents a kernel generation function based on bilinear interpolation, which ensures consistent modeling capabilities across different scales without introducing additional scale-specific parameters; Secondly, based on the instantiated directional convolution kernels, the LADP module performs a two-stage directional decomposition modeling. In the first stage, vertical and horizontal convolution kernels are applied to their respective subspaces to capture deformation cues along a single dimension, expressed as follows: ; ; in, Indicates the number of groups Depth-separable convolution, This represents the GELU activation function; In the second stage, cross-directional thinning is performed by exchanging the directional convolution kernels between the two subspaces, as expressed by: ; ; This mechanism enables the model to integrate orthogonal spatial responses, effectively capturing non-rigid deformations and complex local textures. Through cross-coupling of vertical and horizontal cues, the LADP module can approximate complex, multi-directional gradient flows that are often ignored by standard fixed-kernel convolutions. Therefore, this module enhances sensitivity to subtle, nonlinear facial muscle movements.

[0025] Finally, the LADP module outputs enhanced local features through residual connections. Its expression is: ; in, Maintaining the same spatial resolution as the input, the residual connections preserve the original orientation features. This represents the splicing of channel dimensions. This decomposition and cross-fertilization strategy effectively captures specific directional deformation cues crucial for recognizing subtle facial expressions. These refined local features provide high-quality, expression-sensitive detail for subsequent hierarchical spatial-frequency co-fusion.

[0026] Step 4: Based on Step 2 and Step 3, the hierarchical spatial-frequency collaborative fusion module HSFC is used to reconstruct discriminative local-global details in the frequency domain through wavelet transform, and adaptive dual-path fusion between spatial representation and frequency representation is performed to obtain fused features.

[0027] In embodiments of the present invention, such as Figure 3 As shown, step 4 includes: To overcome the limitations of spatial dimension-only fusion, the HSFC module adopts a dual-path adaptive learning framework, which aims to integrate spatial semantic feature representation with frequency domain texture cues. The design motivation of the HSFC module comes from prior knowledge: facial expression features have structural sparsity in the frequency domain, where low-frequency components correspond to global semantics and high-frequency components encode local deformations. By combining spectral decomposition, adaptive component selection and cross-domain collaboration, end-to-end spatial-frequency feature modeling is achieved. Step 41, Airspace Path; The spatial fusion path focuses on the efficient aggregation of multi-source features within the spatial domain. At the i-th pyramid level, local features... Global features And the fusion features propagated from the previous level The expression for concatenating along the channel dimension is: ; in, LN indicates concatenation along the channel dimension, and LN indicates layer normalization. Represents the GELU activation function; branch-specific convolutional layers , and The fused features, local features, and global features propagated from the previous level are projected into a unified embedding space. and Perform channel compression and remapping to ensure spatial blending; Step 42, Frequency Domain Path; The frequency domain path is used to decouple and enhance discriminative features from a spectral perspective. Compared to Fourier transform, discrete wavelet transform (DWT) offers superior spatiotemporal-frequency localization capabilities, performing particularly well in separating non-stationary texture signals in face regions. Therefore, the HSFC module introduces discrete wavelet transform (DWT) for frequency domain modeling in the parallel fusion path, with the expression: ; ; Where LL represents the low-frequency subband, and LH, HL and HH correspond to the high-frequency subbands along different directions, which together constitute the initial spectrum dictionary of facial expressions; Learnable parameters are then introduced. The expression for adaptive fusion across frequency bands is: ; These parameters enable the model to adaptively optimize the combination ratio of spectral features to meet the needs of local details and global semantics, allowing the model to automatically focus on the texture features that are most discriminative for facial expression recognition and suppress irrelevant interference.

[0028] To enhance the discriminative representation capability of frequency domain sub-bands, a cross-band attention mechanism is introduced to capture the potential correlations and complementary information between different frequency components. In the spectral distribution, the low-frequency sub-band contains the topological structure and global semantics of the image, serving as the core foundation for representing expression categories. A channel attention mechanism is added to the fused low-frequency features, with the following expression: ; in, This indicates element-wise multiplication. This represents the Sigmoid activation function. Two layers for channel excitation The system consists of convolutional layers; this mechanism, through adaptive recalibration of dependencies between channels, can accurately select and enhance feature channels sensitive to facial expression semantics, while suppressing redundant noise in the low-frequency background. This design ensures that the model maintains global robustness while providing high-purity semantic guidance for subsequent hierarchical feature fusion.

[0029] The high-frequency subband is used to capture edge and detail information in an image, exhibiting high spatial locality in facial expressions. Its energy distribution is often concentrated in the local muscle groups that produce facial expressions. Based on this prior knowledge, spatial attention guidance is added to the high-frequency path, and its expression is: ; By learning the spatial weight map, this module can dynamically recalibrate the response intensity of high-frequency features, enabling the model to accurately locate local deformation cues closely related to facial expression semantics even in complex facial environments, thus ensuring the sensitivity and robustness of spatial-frequency co-modeling.

[0030] Subsequently, the refined multi-band signal is reconstructed using the inverse discrete wavelet transform (IDWT) to generate a frequency-domain enhanced feature representation. ; Step 43: Residual Gated Refinement Block (RGRB). To further improve the collaborative efficiency of spatial and frequency paths, a residual gating refinement module RGRB is designed. First, the adaptive dynamic selection of features is achieved through a gating mechanism, and then an inverse residual MLP (IRMLP) is connected. This design not only achieves accurate feature alignment in the initial fusion stage, but also enhances the expressive power of complex expression patterns through high-dimensional manifold mapping, thus constructing a complete inference unit from feature selection to deep reconstruction. To ensure effective integration of spatial semantic consistency and frequency domain accuracy, the HSFC module employs the RGRB module to balance the spatial and frequency domain paths, with the following expression: ; ; This gating strategy enables the model to adaptively select the most discriminative information source based on different facial expression patterns, thereby achieving effective space-frequency coordination.

[0031] Finally, the HSFC module utilizes the inverse residual MLP (IRMLP) to perform nonlinear manifold mapping in a 4-fold expanded high-dimensional space, the expression of which is: ; Here, BN represents batch normalization used to accelerate convergence. This represents the fused features propagated from the previous level. This expansion-compression structure enhances the ability to model complex facial expression patterns while mitigating information loss during fine-grained processing. Through this hierarchical propagation and nonlinear fine-grained processing, fine-grained spatial-frequency information from lower levels is effectively transferred to higher semantic layers, thus ensuring a highly discriminative feature representation for the final classifier.

[0032] Step 5: Based on step 4, project the fused features into a set of feature vectors and input them into a lightweight VIT classifier to generate the final prediction result.

[0033] Facial expression recognition is a fundamental task in affective computing, but it remains challenging due to the subtle expressions caused by low-intensity facial muscle movements. While existing multi-branch networks have shown promise by fusing complementary spatial features, they often neglect frequency domain cues, which are crucial for encoding fine-grained facial details and decoupling subtle discriminative patterns from background noise. This invention proposes a novel pyramid spatial-frequency collaborative framework, PyraWave, for facial expression recognition. PyraWave employs a multi-scale pyramid architecture with a bi-branch design at each scale to capture complementary cues at different representational levels. Specifically, the global branch incorporates a Landmark-guided Global Semantic Modeling (L-GSM) module to capture long-range dependencies in expression-related regions; while the local branch introduces a Local Adaptive Deformation Perception Module (LADP) module, which enhances sensitivity to fine-grained local texture and deformation changes through two-stage directional modeling, aiming to jointly capture global dependencies and local fine-grained muscle deformations. Furthermore, this invention designs a Hierarchical Spatial-Frequency Collaborative (HSFC) fusion module. This module utilizes wavelet-based spectral decomposition and a cross-band attention mechanism to adaptively reconstruct discriminative spectral components. It also designs residual-gated refinement blocks to balance the contributions of space and frequency. By propagating and refining the fusion features from the previous layer, this module achieves cross-scale feature recalibration. Using wavelet-based multi-scale spectral decomposition, it models fine-grained facial deformations while maintaining spatial correspondence, achieving adaptive interaction between spatial semantics and frequency-aware details. Through a hierarchical propagation strategy, information is effectively exchanged and enhanced across different scales. Finally, the highest-level fusion features are projected into a set of feature vectors and input into a lightweight Vision Transformer (VIT) classifier to generate the final prediction result. This architecture enables the model to accurately locate discriminative expression signals amidst complex background noise, thereby enhancing the model's sensitivity to subtle expressions.

[0034] Experiments on widely used facial expression recognition benchmark datasets demonstrate that PyraWave exhibits highly competitive and excellent performance. Ablation studies and visualization analysis further validate the effectiveness of the proposed space-frequency fusion strategy. This work shows that explicit frequency domain modeling can effectively complement spatial domain methods, providing a new perspective on space-frequency co-modeling in facial expression recognition.

[0035] Compared with the prior art, the present invention has the following advantages: The PyraWave framework of this invention adopts a multi-scale pyramid architecture, which can progressively extract and refine facial expression features across different semantic levels. This invention designs a dual-branch feature extraction strategy, which integrates a landmark-guided global semantic modeling module (L-GSM) and a local adaptive deformation perception module (LADP), realizing complementary modeling of global dependencies and local subtle deformations. This invention designs a hierarchical spatial-frequency collaborative fusion module (HSFC), which, by introducing wavelet-based spectral decomposition and cross-band attention mechanism, realizes adaptive interaction between spatial and frequency domain representations, thereby effectively capturing discriminative spectral details.

[0036] The technical solution provided by this invention includes the following steps: given an input facial expression image, extract pyramid features; in the global branch, a global semantic modeling module (L-GSM) is used to guide a global attention mechanism to focus on expression-related regions by introducing facial landmark priors, in order to model long-range dependencies; in the local branch, a local adaptive deformation perception module (LADP) is used to capture texture changes with direction perception capabilities in facial expressions through a kernel instantiation mechanism; a hierarchical spatial-frequency collaborative fusion module (HSFC) is used to reconstruct discriminative local-global details in the frequency domain through wavelet transform, and adaptive dual-path fusion between spatial representation and frequency representation is performed to obtain fused features; the fused features are projected into a set of feature vectors and input into a lightweight VIT classifier to generate the final prediction result. This method, by constructing a multi-scale pyramid architecture, achieves deep collaboration between spatial semantics and frequency domain features, solving the problem of difficulty in capturing subtle expressions in facial expression recognition.

[0037] The various steps in the embodiments of the present invention can be performed by an electronic device. This electronic device includes, but is not limited to, tablet computers, portable PCs, and desktop computers.

[0038] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the electronic device containing the computer-readable storage medium to execute the above-described embodiment of the multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion.

[0039] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 4As shown, the electronic device 21 includes a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the multi-branch pyramid expression recognition method based on hierarchical spatial-frequency fusion in the embodiment. To avoid repetition, it will not be described in detail here.

[0040] Electronic device 21 includes, but is not limited to, processor 211 and memory 212. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 21 and does not constitute a limitation on electronic device 21. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0041] The processor 211 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0042] The memory 212 can be an internal storage unit of the electronic device 21, such as a hard disk or RAM of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the electronic device 21. Furthermore, the memory 212 can include both internal and external storage units of the electronic device 21. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 can also be used to temporarily store data that has been output or will be output.

[0043] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion, characterized in that, The method includes: Step 1: Given an input image of a human facial expression. Extracting pyramid features , Let C represent the feature at the i-th scale, and let C, H, and W represent the channel, image height, and width, respectively. Step 2: Based on Step 1, the global semantic modeling module L-GSM is used in the global branch. By introducing facial landmark priors, the global attention mechanism is guided to focus on expression-related regions in order to model remote dependencies. Step 3: Based on Step 1, in the local branch, the Local Adaptive Deformation Perception Module (LADP) is used to capture texture changes with direction perception capabilities in facial expressions through the kernel instantiation mechanism. Step 4: Based on Step 2 and Step 3, the hierarchical spatial-frequency collaborative fusion module HSFC is used to reconstruct discriminative local-global details in the frequency domain through wavelet transform, and adaptive dual-path fusion between spatial representation and frequency representation is performed to obtain fused features; Step 5: Based on step 4, project the fused features into a set of feature vectors and input them into the lightweight VIT classifier to generate the final prediction result. Step 4 includes: The HSFC module adopts a dual-path adaptive learning framework, which aims to integrate spatial semantic feature representation with frequency domain texture cues. The design motivation of the HSFC module comes from prior knowledge: facial expression features have structural sparsity in the frequency domain, where low-frequency components correspond to global semantics and high-frequency components encode local deformations. By combining spectral decomposition, adaptive component selection and cross-domain collaboration, end-to-end spatial-frequency feature modeling is achieved. Step 41, Airspace Path; The spatial fusion path focuses on the efficient aggregation of multi-source features within the spatial domain. At the i-th pyramid level, local features... Global features And the fusion features propagated from the previous level The expression for concatenating along the channel dimension is: ； in, LN indicates concatenation along the channel dimension, and LN indicates layer normalization. Represents the GELU activation function; branch-specific convolutional layers , and The fused features, local features, and global features propagated from the previous level are projected into a unified embedding space. and Perform channel compression and remapping to ensure spatial blending; Step 42, Frequency Domain Path; The frequency domain path is used to decouple and enhance discriminative features from a spectral perspective. The HSFC module introduces Discrete Wavelet Transform (DWT) for frequency domain modeling in the parallel fusion path, and its expression is as follows: ； ； in, Let represent the discrete wavelet transform function, LL represent the low-frequency subband, and LH, HL and HH correspond to the high-frequency subbands along different directions, which together constitute the initial spectral dictionary of facial expressions; Learnable parameters are then introduced. The expression for adaptive fusion across frequency bands is: ； To enhance the discriminative representation capability of frequency domain subbands, a cross-band attention mechanism is introduced to capture the potential correlations and complementary information between different frequency components. In the spectral distribution, the low-frequency subband contains the image's topological structure and global semantics. A channel attention mechanism is added to the fused low-frequency features, with the expression: ； in, This indicates element-wise multiplication. This represents the Sigmoid activation function. Two layers for channel excitation Convolutional layer composition; The high-frequency subband is used to capture edge and detail information of the image. Spatial attention guidance is added to the high-frequency path, and its expression is: ； Subsequently, the refined multi-band signal is reconstructed using the inverse discrete wavelet transform (IDWT) to generate a frequency-domain enhanced feature representation. ; Step 43: Residual Gating Refinement Module RGRB; The residual gating refinement module RGRB is designed. First, the adaptive dynamic selection of features is realized through the gating mechanism. Then, an inverse residual IRMLP is connected to build a complete inference unit from feature selection to deep reconstruction. To ensure effective integration of spatial semantic consistency and frequency domain accuracy, the HSFC module employs the RGRB module to balance the spatial and frequency domain paths, with the following expression: ； ； in, Indicates the gating mechanism function; Finally, the HSFC module utilizes the inverse residual IRMLP to perform nonlinear manifold mapping in a 4-fold expanded high-dimensional space, the expression of which is: ； Where BN represents batch normalization used to accelerate convergence. This indicates a fusion feature that has been propagated from the previous level.

2. The method according to claim 1, characterized in that, Step 2 includes: First, a pre-trained and frozen landmark feature extractor is used to generate landmark-aware semantic features from the input facial expression image. Landmark-aware semantic features are used to encode the spatial distribution of expression-related key regions and serve as structural guidance information for global branch modeling. In each level of the pyramid, landmark-aware semantic features are linearly projected to form query vectors, while the corresponding backbone network features are projected to generate key and value vectors. Driven by queries guided by landmark information, global contextual information is integrated through aggregation based on a global attention mechanism. To enhance spatial awareness, a learnable relative position bias is incorporated into the global attention calculation process, the expression of which is: ； in, B represents the feature dimension and B represents the learnable relative position bias, which is used to supplement spatial position perception ability. Global attention output The final global features are obtained by refining them using a feedforward network. .

3. The method according to claim 2, characterized in that, Step 3 includes: First, the pyramid features input at stage i are divided into two subspaces along the channel dimension: ,in These are used to model local spatial dependencies in the vertical and horizontal directions, respectively; for a specific spatial scale... The LADP module generates functions through a lightweight kernel. From shared kernel parameters A set of instance-aware convolutional kernels is derived, and their expressions are as follows: ； in, Represents a kernel generating function based on bilinear interpolation; Secondly, based on the instantiated directional convolution kernels, the LADP module performs a two-stage directional decomposition modeling. In the first stage, vertical and horizontal convolution kernels are applied to their respective subspaces to capture deformation cues along a single dimension, as expressed in the following expression: ； ； in, Indicates the number of groups Depth-separable convolution, This represents the GELU activation function; In the second stage, cross-directional thinning is performed by exchanging the directional convolution kernels between the two subspaces, as expressed by: ； ； Finally, the LADP module outputs enhanced local features through residual connections. Its expression is: ； in, Maintaining the same spatial resolution as the input, the residual connections preserve the original orientation features. This indicates the splicing of channel dimensions.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion as described in any one of claims 1 to 3.

5. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the multi-branch pyramid facial expression recognition method based on hierarchical spatial-frequency fusion as described in any one of claims 1 to 3.

Images