A face attribute recognition method based on a state space model
By combining the ConvNeXt network with the feature enhancement module of the state space model, the problems of insufficient global modeling of CNN and high computational complexity of Transformer in the existing technology are solved, realizing efficient and accurate face attribute recognition, which is suitable for edge devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUIZHOU UNIV
- Filing Date
- 2026-03-31
- Publication Date
- 2026-07-10
AI Technical Summary
In existing technologies, CNN-based methods have strong local feature extraction capabilities in face attribute recognition but lack global contextual information. Although attention mechanisms have improved this, they have high computational complexity. Transformer-based methods, while having strong global relationship modeling capabilities, have large model sizes and high computational resource consumption, making them difficult to apply in real time on resource-constrained platforms.
The ConvNeXt network, combined with a state-space model, is used to combine local fine-grained features with long-distance dependencies through local feature extraction, feature fusion, feature enhancement, and classification modules. Lightweight feature enhancement is achieved using the state-space model, and multi-task learning and an appropriate loss function are employed for training.
It achieves high recognition accuracy while reducing model complexity, is suitable for edge devices with limited computing resources, can accurately classify facial expressions and multiple facial attributes, and improves the model's generalization ability and computational efficiency.
Smart Images
Figure CN122369078A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and pattern recognition technology, and specifically to a face attribute recognition method based on a state-space model. Background Technology
[0002] Facial attribute recognition aims to extract human-understandable visual characteristics from facial images, such as age, gender, facial expressions, and jewelry wearing. It is a fundamental task in the field of computer vision and has wide applications in intelligent surveillance, human-computer interaction, and social entertainment. With the rapid development of deep learning technology, methods based on convolutional neural networks (CNNs) and Transformers have become the mainstream technical approach for facial attribute recognition.
[0003] Traditional CNN-based methods, such as ResNet and ConvNeXt, extract hierarchical features from images through local receptive fields, excelling in capturing fine-grained local facial features (such as facial details and texture information). For example, Li Ya et al. proposed a network LFaceA based on lightweight residual modules and multi-task learning ("A Face Attribute Recognition Method Based on Multi-Task Learning," published in *Computer Engineering*, 2020, Vol. 46, No. 3). This network reduces the number of parameters through depthwise separable convolutions and designs shared branch networks based on attribute relationships, improving recognition efficiency while achieving lightweight design. However, CNN methods are inherently limited by their local computational characteristics, having a natural weakness in modeling the relationships between distant pixels in an image (i.e., global contextual information). This may lead to performance limitations in tasks involving overall facial layout or complex relationships between attributes.
[0004] To compensate for the shortcomings of CNNs in global feature modeling, attention mechanisms have been widely introduced. For example, the FAER model proposed by Xu Jie et al. ("Face Attribute Estimation and Expression Recognition Based on Contextual Channel Attention Mechanism," published in *Computer Applications*, 2025, Vol. 45, No. 1) uses ConvNeXt as the backbone network to extract local features and designs a contextual channel attention (CC Attention) mechanism. By dynamically adjusting channel weights, it learns global and local features, effectively enhancing the model's ability to represent contextual information and achieving excellent performance in face attribute estimation and expression recognition tasks. Although attention mechanisms can model global dependencies to a certain extent, they are usually based on explicit correlation calculations of pixel pairs or channel pairs, leaving room for optimization in terms of computational complexity.
[0005] In recent years, Transformer-based models, such as Vision Transformer (ViT) and its variants, have shown potential in face attribute recognition tasks due to their powerful global relation modeling capabilities. For example, SwinFace (published in IEEE Transactions on Circuits and Systems for Video Technology, Vol. 34, No. 4, 2024) can simultaneously handle multiple tasks such as recognition, expression recognition, and age estimation through a shared backbone network and multi-task design. However, Transformer models typically rely on self-attention mechanisms, and their computational complexity is proportional to the square of the input sequence length, resulting in a large number of model parameters and high computational resource requirements. This limits their deployment and application on mobile or embedded devices with limited computing resources.
[0006] In summary, the existing technology mainly has the following problems and shortcomings:
[0007] CNN-based methods, while adept at extracting local features, are insufficient in modeling global contextual information and long-distance dependencies, which may affect the identification of complex relationships between attributes.
[0008] The method of incorporating attention mechanisms can, to some extent, make up for the shortcomings of global modeling in CNNs, but the computational overhead of attention mechanisms may still increase the complexity of the model.
[0009] Transformer-based methods, while possessing powerful global relationship modeling capabilities, generally suffer from large model sizes and high computational resource consumption, making it difficult to meet the real-time application needs on resource-constrained platforms.
[0010] Therefore, a model architecture is needed that can combine the efficiency of CNNs in extracting local fine-grained features with the ability of Transformers or similar models to capture long-range dependencies, while avoiding the computational burden caused by either extreme, thereby achieving a balance between accuracy and efficiency. Summary of the Invention
[0011] The purpose of this invention is to solve the above-mentioned problems by providing a face attribute recognition method based on a state space model that reduces model complexity while ensuring recognition accuracy, and achieves accurate and efficient classification of facial expressions and various facial attributes.
[0012] To achieve the above objectives, the technical solution adopted by the present invention is summarized as follows:
[0013] The present invention provides a face attribute recognition method based on a state-space model, comprising the following steps:
[0014] Step 1: Obtain a face image dataset and preprocess the images. The dataset contains face attribute labels and / or face expression labels. Step 2: Construct a multi-task learning network, the network comprising:
[0015] The local feature extraction module uses a ConvNeXt network structure to extract multi-scale local fine-grained feature maps from the input face image;
[0016] The feature fusion module, connected to the local feature extraction module, is used to fuse the multi-scale local fine-grained feature maps to obtain a fused feature map.
[0017] The feature enhancement module, connected to the feature fusion module, is used to convert the fused feature map into a one-dimensional sequence, and to use a pre-trained selective scanning algorithm based on a state space model to model the long-distance dependency relationship of the one-dimensional sequence, and output the enhanced feature sequence.
[0018] The classification module, connected to the feature enhancement module, is used to map the enhanced feature sequence to the final face attribute category probability distribution. Step 3: Jointly train the multi-task learning network using the face image dataset to optimize the network parameters; Step 4: Input the face image to be recognized into the trained multi-task learning network to obtain the face attribute recognition result.
[0019] The aforementioned face attribute recognition method based on a state-space model, wherein the local feature extraction module is a truncated ConvNeXt network, specifically, the part after the last stage of the ConvNeXt network is removed, which is used to progressively encode the input feature map into a 14×14 feature map.
[0020] The aforementioned face attribute recognition method based on a state-space model, wherein the feature fusion module includes:
[0021] The first convolutional block consists of a two-dimensional convolutional layer with a kernel size of 3×3, and is used to perform channel mixing on the feature map output by the local feature extraction module.
[0022] The second convolutional block, consisting of two-dimensional convolutional layers with a kernel size of 1×1, is used to extract multi-scale features;
[0023] The feature overlay unit is used to add the output feature maps of the first convolutional block and the second convolutional block element by element to achieve feature fusion.
[0024] The aforementioned face attribute recognition method based on a state-space model includes a feature enhancement module comprising an embedding layer and N stacked feature enhancement blocks based on the state-space model. The embedding layer is used to convert the fused feature map into a one-dimensional sequence tensor that meets the input requirements of the state-space model. Each feature enhancement block contains a state-space model unit, which processes the input sequence tensor through discretized state update equations and output equations to capture long-distance dependencies between sequences.
[0025] The aforementioned face attribute recognition method based on a state-space model, wherein the processing procedure of the state-space model unit is characterized by the following equation:
[0026] State update equation:
[0027] Output equation:
[0028] in, Let be the input sequence at time t. Let be the hidden state vector at time t. The output at time t, It is a dynamic parameter matrix related to the input.
[0029] The aforementioned face attribute recognition method based on a state-space model includes a classification module comprising two parallel classifiers for face attribute estimation and facial expression recognition tasks, respectively. The face attribute estimation task employs Focal Loss as the loss function to address the imbalanced sample problem, while the facial expression recognition task employs cross-entropy loss.
[0030] Compared with the prior art, the present invention has significant advantages, as can be seen from the above technical solution:
[0031] 1. Balancing Local and Global Features: This invention combines the ConvNeXt network with a feature enhancement module based on a state-space model. ConvNeXt is responsible for extracting rich local fine-grained features, while the state-space model can effectively capture long-distance dependencies in sequence data, thereby compensating for the shortcomings of the receptive field of CNNs and enabling the model to make more accurate judgments by utilizing both local texture and global structural information.
[0032] 2. Lightweight Model and High Computational Efficiency: The state-space model exhibits near-linear scalability when processing long sequences, and its computational complexity and parameter count are lower compared to the Transformer's self-attention mechanism. This allows the overall model of this invention to maintain high accuracy while being more lightweight, making it easier to deploy on edge devices with limited computing resources.
[0033] 3. Strong multi-task learning capability: This invention adopts a unified network architecture, sharing underlying features and designing separate classifiers and loss functions for facial attribute estimation and expression recognition tasks, thus achieving multi-task joint learning. This mechanism not only simplifies model deployment but also leverages the correlation between different tasks to improve the model's generalization ability and overall performance. Attached Figure Description
[0034] Figure 1 This is a schematic diagram of the network structure of the present invention.
[0035] Figure 2 This is a visualization comparison of the results of this invention on the AffectNet dataset with the baseline model. Detailed Implementation
[0036] The following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the face attribute recognition method based on a state-space model proposed in this invention:
[0037] A face attribute recognition method based on a state-space model includes the following steps:
[0038] Step 1: Data Preparation and Preprocessing
[0039] The publicly available datasets CelebA (for face attribute estimation) and RAF-DB (for facial expression recognition) were used for model training and evaluation. All images were uniformly scaled to 224×224 pixels. During the training phase, data augmentation strategies such as random horizontal flipping (probability 0.5) and random erasure (probability 0.5) were applied to the images to improve the model's generalization ability. The CelebA dataset contains 162,770 training images and 19,962 test images, with each image labeled with 40 binary attributes. The RAF-DB dataset contains 12,271 training images and 3,068 test images, labeled with 7 basic emotions.
[0040] Step 2: Construct the SSM-FAR network
[0041] See Figure 1 The SSM-FAR network mainly consists of four parts: a local feature extraction module, a feature fusion module, a feature enhancement module, and a classification module.
[0042] Step 2.1: Local Feature Extraction Module
[0043] The ConvNeXt-S model (ConvNeXt-S for short), pre-trained on ImageNet, serves as the backbone network. After removing its last stage (i.e., Stage 4), the original ConvNeXt-S input image size is 224×224. After convolution and downsampling from Stage 1 to Stage 3, the spatial size of the output feature map becomes 14×14 with 512 channels. The output of this module is denoted as f0 (size 14×14, number of channels 512). f0 contains rich fine-grained information such as local texture, edges, and shape.
[0044] Step 2.2: Feature Fusion Module
[0045] To further enrich the scale information of the features, a feature fusion module is adopted. The input of this module is f0, and the processing flow is as follows:
[0046] First, f0 passes through another 2D convolutional layer with a kernel size of 3×3 (Conv2D 3×3). Padding (1) maintains the spatial size, allowing for the extraction of multi-scale features with a slightly larger receptive field, resulting in feature map f1 (14×14, 512 channels). f1 then passes through another 2D convolutional layer with a kernel size of 1×1 (Conv2D 1×1). While maintaining the spatial size (7×7), information exchange and compression are performed on the 512 channels, resulting in feature map f2 (512 channels, 7×7 in this embodiment). Finally, f1 and f2 are added element-wise to obtain feature map f that incorporates multi-scale information. fused (Size is 7×7, number of channels is 512):
[0047] f fused =f1 +f2
[0048] Step 2.3: Feature Enhancement Module
[0049] The core of the feature enhancement module is a stack of N state-space model-based feature enhancement blocks (SSM Blocks), N=12; first, the feature map f fused The sequence is flattened and mapped into a one-dimensional sequence tensor x (sequence length 49, embedding dimension 512) through an embedding layer; then, x passes through 12 SSM blocks, the internal structure and specific parameter settings of each SSM block are as follows:
[0050] 1) For the input sequence xi Perform RMSNorm normalization;
[0051] 2) The normalized sequence is passed through a linear projection layer to expand the dimension from 512 to 1024 (i.e., the expansion factor is set to 2), and then passed through a one-dimensional convolution with a kernel size of 4 and the SiLU activation function to obtain the processed sequence;
[0052] 3) The processed sequence is input into the core State-Space Model (SSM) unit, with the state dimension L of the SSM unit set to 16. Using a selective scan algorithm, the sequence is modeled using the following discretized state update equations and output equations to capture long-distance dependencies:
[0053]
[0054] in, Let be the input sequence at time t. Let be the hidden state vector at time t. The output at time t, It is a dynamic parameter matrix related to the input, ensuring that the model can adaptively adjust according to different input content;
[0055] 4) The output of the SSM unit passes through a linear projection layer (restoring the dimension from 1024 to 512), and then is residually concatenated with X, which has also passed through a linear projection layer, to obtain the output of the SSM Block. After stacking 12 blocks, the final feature sequence x, which enhances the global context information, is obtained. N (The sequence length is 49, and the embedding dimension is 512). Meanwhile, for subsequent classification tasks, X... out Perform an RMSnorm normalization, a LayerNorm normalization, and a Softmax activation function dimensionality transformation to reconstruct the feature map X (size 14×14, number of channels 512).
[0056] Step 2.4: Classification Module
[0057] This module contains two parallel classifiers, each corresponding to a different task;
[0058] Facial expression recognition branch: X is input to a global average pooling layer, resulting in a 512-dimensional feature vector. This vector is then input to a fully connected layer, outputting a 7-dimensional logits vector corresponding to 7 expression categories. This branch uses the cross-entropy loss function L. CE Conduct supervised training;
[0059] Face attribute estimation branch: Similarly, X is input to another global average pooling layer and a fully connected layer, outputting a 40-dimensional logits vector corresponding to 40 binary attributes. Due to the potentially severe imbalance between positive and negative samples in the face attribute dataset, this branch employs Focal Loss L... FL As a loss function, Focal Loss introduces a moderating factor to make the model pay more attention to samples that are difficult to classify. Its calculation formula is as follows:
[0060]
[0061] in, It is the model's predicted probability of the true class. This is the focus parameter (set to 2). It is a category weight parameter;
[0062] Step 3: Model Training and Parameter Optimization
[0063] The AdamW optimizer is used to perform end-to-end joint training of the network. First, the pre-trained weights of ConvNeXt-S on ImageNet are loaded to initialize the local feature extraction module, and the weights of other modules are initialized using Xavier.
[0064] The training parameters are set as follows:
[0065] Batch size: Set to 100 for the CelebA dataset and 110 for the RAF-DB dataset.
[0066] Initial learning rate: Set to 2.5 × 10 for CelebA. -4 For RAF-DB, set to 4×10 -3 ;
[0067] Learning rate decay strategy: ExponentialLR is used, and the decay factor Gamma is set to 0.95;
[0068] Training epochs: 100 epochs on CelebA and 120 epochs on RAF-DB;
[0069] The total loss function is the sum of the losses from each task: L total =L CE +L FL The backpropagation algorithm is used to update all parameters of the entire network.
[0070] Step 4: Facial Attribute Recognition
[0071] The 224×224 face image to be identified is input into the trained SSM-FAR network. After forward propagation, the predicted probabilities of 40 attributes (usually output by the Sigmoid function) and the predicted probabilities of 7 expression categories (output by the Softmax function) are obtained from the classification module. The final face attribute recognition result is obtained by taking the category with the highest probability or judging according to the threshold.
[0072] The method in this embodiment can simultaneously recognize 40 binary facial attributes (such as gender, whether or not glasses are worn) and 7 basic facial expressions (anger, disgust, fear, happiness, sadness, surprise, and neutral). On the CelebA and RAF-DB test sets, the average accuracy reached 91.70% and 91.23% respectively, outperforming existing technologies. Visualization results (such as...) Figure 2 As shown in the figure, this method can more accurately focus on key facial areas and effectively distinguish subtle differences between different expressions.
[0073] The above are merely preferred embodiments of the present invention and do not impose any limitations on the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from any technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.
Claims
1. A face attribute recognition method based on a state-space model, characterized in that, Includes the following steps: Step 1: Obtain a face image dataset and preprocess the images. The dataset contains face attribute labels and / or face expression labels. Step 2: Construct a multi-task learning network, the network comprising: The local feature extraction module uses a ConvNeXt network structure to extract multi-scale local fine-grained feature maps from the input face image; The feature fusion module, connected to the local feature extraction module, is used to fuse the multi-scale local fine-grained feature maps to obtain a fused feature map. The feature enhancement module, connected to the feature fusion module, is used to convert the fused feature map into a one-dimensional sequence, and to use a pre-trained selective scanning algorithm based on a state space model to model the long-distance dependency relationship of the one-dimensional sequence, and output the enhanced feature sequence. A classification module, connected to the feature enhancement module, is used to map the enhanced feature sequence to the final face attribute category probability distribution; Step 3: Jointly train the multi-task learning network using the face image dataset to optimize the network parameters; Step 4: Input the face image to be recognized into the trained multi-task learning network to obtain the face attribute recognition result.
2. The face attribute recognition method based on a state-space model according to claim 1, characterized in that, The local feature extraction module is a truncated ConvNeXt network, specifically, the part after the last stage of the ConvNeXt network is removed.
3. The face attribute recognition method based on a state-space model according to claim 1, characterized in that, The feature fusion module includes: The first convolutional block consists of a two-dimensional convolutional layer with a kernel size of 3×3, and is used to perform channel mixing on the feature map output by the local feature extraction module. The second convolutional block, consisting of two-dimensional convolutional layers with a kernel size of 1×1, is used to extract multi-scale features; The feature overlay unit is used to add the output feature maps of the first convolutional block and the second convolutional block element by element to achieve feature fusion.
4. The face attribute recognition method based on a state-space model according to claim 1, characterized in that, The feature enhancement module includes an embedding layer and N stacked feature enhancement blocks based on a state-space model. The embedding layer is used to convert the fused feature map into a one-dimensional sequence tensor that meets the input requirements of the state-space model. Each feature enhancement block contains a state-space model unit, which processes the input sequence tensor through discretized state update equations and output equations to capture long-distance dependencies between sequences.
5. The face attribute recognition method based on a state-space model according to claim 4, characterized in that, The processing of the state-space model unit is characterized by the following equation: State update equation: ; Output equation: ; in, Let be the input sequence at time t. Let be the hidden state vector at time t. The output at time t, It is a dynamic parameter matrix related to the input.
6. The face attribute recognition method based on a state-space model according to claim 1, characterized in that, The classification module includes two parallel classifiers, one for face attribute estimation and the other for facial expression recognition. The face attribute estimation task uses Focal Loss as the loss function, while the facial expression recognition task uses cross-entropy loss.