A fixed-pose expression recognition method of sliding surface patch attention mechanism

Abstract

The application discloses a fixed posture expression recognition method of a sliding surface patch attention mechanism, and comprises the following steps: S1, data preprocessing and deep feature extraction: collecting a fixed posture facial expression dataset and preprocessing, adopting a Conv_1 to Conv_4 convolution module of a ResNet50 convolutional neural network to extract a deep convolution feature map, and normalizing the size of the deep convolution feature map to wherein 14*14 and 1024 respectively represent the spatial and channel domain size of the feature map, and the application has the following advantages: the sliding surface patch (SP) module designed in the application does not need to position feature points of an original picture and pre-divide local surface patches according to experience, but only generates the sliding surface patch on the deep convolution feature map through preset parameters and constraint conditions, thereby breaking the dependence on the positioning accuracy of feature points and the experience of researchers, solving the problems of traditional coordinate surface patch positioning drift and poor universality of segmented surface patches, and avoiding the defect of isolated local features through surface patch overlapping design.

Description

Technical Field

[0001] This invention relates to the field of facial expression recognition methods, specifically a fixed-pose facial expression recognition method based on a sliding patch attention mechanism. Background Technology

[0002] "Fixed-pose facial expression recognition is an important research direction in the field of computer vision, and it is widely used in many fields such as human-computer interaction, intelligent monitoring, psychological assessment, and medical assistance. Traditional fixed-pose facial expression recognition methods need to extract facial texture, geometric or mixed feature attributes to enhance the ability to represent facial images from different viewing angles. Among them, the mainstream feature acquisition methods are divided into two types: coordinate feature patches (LP) and segmentation feature patches (DP).

[0003] Coordinate feature patches (LP) require a set of feature points to be preset in the eye, eyebrow, mouth and cheek areas, and then the facial features to be cropped around the feature points. This can effectively reduce redundant facial features and accurately represent facial semantic information. However, this method relies heavily on the accurate positioning of feature points. Under extreme viewing angles, facial coordinates are prone to inaccurate positioning and drift, resulting in a decrease in recognition accuracy.

[0004] Feature segmentation patches (DP) divide a facial image into a set of regular or irregular local patches, then extract features such as LBP, LPP, SIFT, and HOG and feed them into a fixed pose classifier for expression classification. This can ensure the integrity of the original image information and reduce the sensitivity of facial expression recognition to some extent to pose changes and self-occlusion. However, the method of segmenting local patches depends on the researcher's experience and requires detailed analysis and design of facial images. It has poor versatility and is difficult to adapt to complex scenarios.

[0005] While existing deep learning methods, such as convolutional neural network-based facial expression recognition models, can extract deep texture features, they do not fully consider the optimal selection of local patches and the synergistic enhancement of channel and spatial domain features. In extreme viewpoints, facial occlusion, and complex outdoor environments, they still suffer from insufficient recognition accuracy, high model complexity, and weak generalization ability. Therefore, there is an urgent need for a fixed-pose facial expression recognition method that can break free from the dependence on feature point localization and empirical patch segmentation, while simultaneously achieving synergistic enhancement of channel and spatial domain features, in order to improve recognition performance in complex scenes. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing a fixed-pose facial expression recognition method based on a sliding facet attention mechanism.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A fixed-pose facial expression recognition method based on a sliding facet attention mechanism includes the following steps: S1: Data preprocessing and deep feature extraction: A fixed-pose facial expression dataset is collected and preprocessed. Deep convolutional feature maps are extracted using Conv_1 to Conv_4 convolutional modules of a ResNet50 convolutional neural network. The size of the deep convolutional feature maps is normalized to... Where 14×14 and 1024 represent the spatial and channel domain dimensions of the feature map, respectively; S2: The sliding patch (SP) module generates local patches: In order to ensure that the sliding patch can accurately crop patch feature maps at different positions in the deep feature map, the SP model needs to satisfy the following constraints:

[0009]

[0010] in To obtain the final number of sliding patches in the SP model, The spatial dimension of the deep feature map. For positive integer local slices, and These represent the side length and step size of the sliding window in the deep feature map, respectively. This corresponds to the initial position of the sliding surface. For local slices along The sliding direction of the shaft, The number of sliding patches generated in the feature map, and the parameter and Meet the conditions This ensures that features between sliding patches maintain partial overlap, thus avoiding the defect of isolated features between traditional local patches. Local patch size... and sliding step size The settings can be configured as needed; S3: Channel Domain Attention Mechanism (SE) module processing: Each sliding face is fed into the bottleneck layer of the ResNet50 network, and the SE module is embedded between the two intermediate convolutional layers of the bottleneck layer; the squeezing operation, activation operation, and channel multiplication operation are performed in sequence to obtain the channel domain filtered feature map. S4: Spatial Domain Attention Mechanism (SPA) module processing: Each channel domain filtered feature map is fed into the SPA module, which contains two branches: maintenance channel and attention channel; the maintenance channel generates spatial domain feature vectors, and the attention channel generates weight vectors. The spatial domain feature vectors and weight vectors are multiplied element-wise to obtain the spatial domain weighted feature map; S5: Feature fusion and optimal parameter selection: All spatial domain weighted feature maps are integrated to obtain a global feature vector, which is then fed into a fully connected layer and a classifier to train the expression classification model; the spatial dimension parameter and stride parameter are adjusted, and multiple sets of experiments are conducted on a controlled environment dataset to select the optimal parameter combination;

[0011] S6: Fixed-pose facial expression recognition: A ResNet50 (SPA-SE) facial expression recognition network is constructed using the optimal parameter combination. The fixed-pose facial expression image to be recognized is processed according to the preprocessing method in step 1 and then input into the network. Deep feature extraction, sliding patch generation, channel domain attention filtering, spatial domain attention extraction, and feature fusion are completed in sequence. Finally, the facial expression recognition result is output through the classifier.

[0012] Preferably, in S1, the fixed-pose facial expression dataset includes a controlled environment dataset (BU-3DFEP1, Multi-PIEP1) and an outdoor environment dataset (Pose-RAF-DB, Pose-AffectNet), covering extreme observation angles of 0°, 15°, 30°, 45°, 60°, 75°, and 90°, as well as six basic facial expressions: anger, disgust, happiness, fear, sadness, and surprise. Some datasets contain facial occlusion samples.

[0013] The ResNet50 network is pre-trained on the ImageNet dataset to increase the representativeness of the network parameters and avoid overfitting due to insufficient data.

[0014] Preferably, in S2, the dimensions of all eligible patches are then calculated using a formula. and step length The optimal parameter combination is , At this point, the ResNet50 (SPA-SE) network achieved a recognition accuracy of 78.01% on the BU-3DFEP1 dataset and 86.77% on the Multi-PIEP1 dataset.

[0015] Preferably, in S3, the squeezing operation compresses the features of each channel into a scalar through global mean pooling. The squeezing operation formula is: ,in and These represent the spatial feature dimensions of the input sliding patch in the bottleneck layer. This represents the output feature map;

[0016] The activation operation is performed using a global average pooling layer and two fully connected layers. The activation operation formula is as follows: ,in and This represents the weight matrix in two fully connected layers; Represents a linear activation function; This represents the Sigmoid activation function; This represents the reduction ratio of the dimension of the weight matrix.

[0017] The output weight feature map This can be expressed as: ,in Indicates the salient features of the output channel domain. Represents a scalar parameter matrix and sliding face Perform a channel multiplication operation.

[0018] Preferably, in S3, the bottleneck layer of the ResNet50 network contains four convolutional layers. The number of channels in the input and output convolutional layers is four times that of the intermediate layers. The number of channels in the input and output convolutional layers is 1024, and the number of channels in the two intermediate convolutional layers is 256. The SE module is embedded between the two intermediate convolutional layers. In the activation operation, the first fully connected layer compresses the channel feature vector to 256 dimensions (reduction ratio r=4), and the second fully connected layer restores it to 1024 dimensions.

[0019] Preferably, the process of maintaining the channels is as follows: the feature map after filtering each channel domain is sequentially passed through a max pooling layer and a fully connected layer to generate a spatial domain feature vector. The processing procedure for the attention channels is as follows: the feature maps after filtering each channel domain are sequentially passed through a max pooling layer and two fully connected layers to generate a weight vector. ;

[0020] The maximum pooling layer and It was set to 3 and 2, where and The kernel size and stride of the max pooling layers are set to 1, respectively. Then, two fully connected layers are used to represent the weight vectors of the local patches, with the feature vector dimensions set to 64 and 1, respectively.

[0021] Preferably, the feature integration first compresses the weighted feature map of each spatial domain through global mean pooling, and then concatenates all feature vectors to obtain a global feature vector;

[0022] In the training of the facial expression classification model, five-fold cross-validation is used on the controlled environment dataset, and the recognition accuracy is verified on the outdoor environment dataset using a predetermined training set and test set.

[0023] The selection criteria for the optimal parameter combination are the highest recognition accuracy and reasonable model complexity.

[0024] Preferably, in S6, the model parameters of the ResNet50 (SPA-SE) facial expression recognition network are 22.7M; on the Pose-RAF-DB dataset, the recognition accuracy is 86.76% when the head pose is >30° and 85.92% when the head pose is >45°; on the Pose-AffectNet dataset, the recognition accuracy is 59.86% when the head pose is >30° and 60.37% when the head pose is >45°.

[0025] Preferably, the SPA-SE module (composed of a sliding patch SP module, a channel domain attention SE module, and a spatial domain attention SPA module) can be embedded in a ResNet18 or ResNet34 convolutional neural network to achieve flexible adaptation between model complexity and recognition accuracy. Specifically, the ResNet18 (SPA-SE) network has a recognition accuracy of 77.64% on the BU-3DFEP1 dataset with 11.2M model parameters; the ResNet34 (SPA-SE) network has a recognition accuracy of 77.76% on the BU-3DFEP1 dataset.

[0026] The beneficial effects of this invention are: the fixed-pose facial expression recognition method based on a sliding patch attention mechanism provided by this invention...

[0027] 1. The sliding patch (SP) module designed in this invention does not require feature point localization and empirical local patch pre-division of the original image. It automatically generates sliding patches on the deep convolutional feature map by means of preset parameters and constraints, thus getting rid of the dependence on feature point localization accuracy and researcher experience. It solves the problems of traditional coordinate patch localization drift and poor versatility of segmented patches. At the same time, through the patch overlapping design, it avoids the defect of isolated local features.

[0028] 2. The Spatial-Channel Attention Mechanism (SPA-SE) module designed in this invention integrates channel domain attention (SE) and spatial domain attention (SPA) into the sliding patch module. It extracts salient facial features from both channel and spatial dimensions, enhances the importance of local patch features, suppresses the influence of redundant features and interference information (such as pose, occlusion, background, and illumination), and significantly improves the accuracy of expression recognition in complex scenes. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the basic structure of the present invention;

[0030] Figure 2 For different dimensions of the present invention ( , A diagram illustrating the process of generating a sliding patch;

[0031] Figure 3This is a diagram of the spatial domain sliding patch attention module of the present invention;

[0032] Figure 4 This is a flowchart of the optimal channel-space sliding surface screening process of the present invention;

[0033] Figure 5 This is a diagram of the channel-spatial domain sliding patch attention module of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0036] Example 1, such as Figure 1 - Figure 5 As shown, the present invention provides a fixed-pose facial expression recognition method based on a sliding facet attention mechanism, comprising the following steps: S1: Data preprocessing and deep feature extraction: A fixed-pose facial expression dataset is collected and preprocessed. Deep convolutional feature maps are extracted using the Conv_1 to Conv_4 convolutional modules of a ResNet50 convolutional neural network. The size of the deep convolutional feature maps is normalized to [size missing]. Where 14×14 and 1024 represent the spatial and channel domain dimensions of the feature map, respectively; S2: The sliding patch (SP) module generates local patches: In order to ensure that the sliding patch can accurately crop patch feature maps at different positions in the deep feature map, the SP model needs to satisfy the following constraints:

[0037]

[0038] in To obtain the final number of sliding patches in the SP model, The spatial dimension of the deep feature map. For positive integer local slices, and These represent the side length and step size of the sliding window in the deep feature map, respectively. This corresponds to the initial position of the sliding surface. For local slices along The sliding direction of the shaft, The number of sliding patches generated in the feature map, and the parameter and Meet the conditions This ensures that features between sliding patches maintain partial overlap, thus avoiding the defect of isolated features between traditional local patches. Local patch size... and sliding step size It can be set as needed;

[0039]

[0040] S3: Channel Domain Attention Mechanism (SE) module processing: Each sliding facet is fed into the bottleneck layer of the ResNet50 network, and the SE module is embedded between the two intermediate convolutional layers of the bottleneck layer; the squeezing operation, activation operation, and channel multiplication operation are performed sequentially to obtain the channel domain filtered feature map. S4: Spatial Domain Attention Mechanism (SPA) module processing: Each channel domain filtered feature map is fed into the SPA module, which contains two branches: a maintenance channel and an attention channel; the maintenance channel generates a spatial domain feature vector, and the attention channel generates a weight vector. The spatial domain feature vector and the weight vector are multiplied element-wise to obtain a spatial domain weighted feature map. S5: Feature Fusion and Optimal Parameter Selection: All spatial domain weighted feature maps are integrated to obtain a global feature vector, which is then fed into a fully connected layer and a classifier to train the facial expression classification model; the spatial dimension parameter and stride parameter are adjusted, and multiple experiments are conducted on a controlled environment dataset to select the optimal parameter combination.

[0041] S6: Fixed-pose facial expression recognition: A ResNet50 (SPA-SE) facial expression recognition network is constructed using the optimal parameter combination. The fixed-pose facial expression image to be recognized is processed according to the preprocessing method in step 1 and then input into the network. Deep feature extraction, sliding patch generation, channel domain attention filtering, spatial domain attention extraction, and feature fusion are completed in sequence. Finally, the facial expression recognition result is output through the classifier.

[0042] In S1, the fixed-pose facial expression dataset includes the controlled environment dataset (BU-3DFEP1, Multi-PIEP1) and the outdoor environment dataset (Pose-RAF-DB, Pose-AffectNet), covering extreme viewing angles of 0°, 15°, 30°, 45°, 60°, 75°, and 90°, as well as six basic facial expressions: anger, disgust, happiness, fear, sadness, and surprise. Some datasets contain facial occlusion samples.

[0043] The ResNet50 network is pre-trained on the ImageNet dataset to increase the representativeness of the network parameters and avoid overfitting due to insufficient data.

[0044] A fixed-pose facial expression dataset was collected, including the controlled environment datasets BU-3DFEP1 and Multi-PIEP1, and the outdoor environment datasets Pose-RAF-DB and Pose-AffectNet. The BU-3DFEP1 dataset contains 5 extreme viewing angles (0°, 30°, 45°, 60°, 90°) and 6 basic expressions, each with 4 exaggeration levels. The Multi-PIEP1 dataset contains 7 extreme viewing angles (0°, 15°, 30°, 45°, 60°, 75°, 90°) and 6 basic expressions. The Pose-RAF-DB dataset contains 1948 training images and 985 test images, while the Pose-AffectNet dataset contains 1248 training images and 558 test images. Both datasets include samples with head poses >30° and >45°, and some samples contain facial occlusion. All facial expression images were preprocessed: image sizes were normalized to obtain standardized input images. A ResNet50 convolutional neural network pre-trained on the ImageNet dataset is used as a deep feature extractor. Conv_1 to Conv_4 convolutional modules are selected, and the standardized input image is input into the module to extract deep convolutional feature maps. The size of the deep convolutional feature maps is normalized to 14×14×1024, where 14×14 is the spatial size and 1024 is the channel domain size.

[0045]

[0046] In S2, the dimensions of all eligible patches are then calculated using a formula. and step length The optimal parameter combination is , At this point, the ResNet50 (SPA-SE) network achieved a recognition accuracy of 78.01% on the BU-3DFEP1 dataset and 86.77% on the Multi-PIEP1 dataset.

[0047] Calculate the number of sliding patches under different parameter combinations based on the constraints to ensure that the sliding patches maintain partial overlap and avoid local feature isolation.

[0048]

[0049] The dimensions and step size of all eligible patches were calculated using the corresponding formula. Through multiple sets of experiments, the optimal parameter combination was determined. , At this point, the ResNet50 (SPA-SE) network achieved a recognition accuracy of 78.01% on the BU-3DFEP1 dataset and 86.77% on the Multi-PIEP1 dataset.

[0050] Based on the optimal parameter combination, the feature map is slid along the x and y axes on a 14×14×1024 deep convolutional feature map to generate corresponding sliding patches, ensuring feature overlap between patches and improving feature integration effect.

[0051] In S3, the squeezing operation compresses the features of each channel into a scalar using global mean pooling. The squeezing operation formula is as follows: (4.2), where and These represent the spatial feature dimensions of the input sliding patch in the bottleneck layer. This represents the output feature map;

[0052] The activation operation employs a global mean pooling layer and two fully connected layers to perform the activation operation. The activation operation formula is as follows: (4.3), where and This represents the weight matrix in two fully connected layers; Represents a linear activation function; This represents the Sigmoid activation function; This represents the reduction ratio of the dimension of the weight matrix.

[0053] Output weight feature map This can be expressed as: (4.4), where Indicates the salient features of the output channel domain. Represents a scalar parameter matrix and sliding face Perform a channel multiplication operation.

[0054] In S3, the bottleneck layer of the ResNet50 network contains four convolutional layers. The input and output convolutional layers have four times the number of channels as the intermediate layers, with a total of 1024 channels for the input and output convolutional layers. The two intermediate convolutional layers have 256 channels each, and the SE module is embedded between the two intermediate convolutional layers. During activation, the first fully connected layer compresses the channel feature vector to 256 dimensions (reduction ratio r=4), and the second fully connected layer restores it to 1024 dimensions.

[0055] Perform a compression operation on each sliding facet: compress the 3×3 features of each channel into a scalar through global mean pooling, resulting in a 1024-dimensional channel feature vector.

[0056] Activation operation: A global mean pooling and two fully connected layers are used. The first fully connected layer compresses the 1024-dimensional channel feature vector to 256 dimensions (reduction ratio r=4). After passing through the ReLU linear activation function, the second fully connected layer restores it to 1024 dimensions, and then the channel weight vector is generated through the Sigmoid activation function. The activation operation formula is as follows: ,in and This represents the weight matrix in two fully connected layers; Represents a linear activation function; This represents the Sigmoid activation function; This represents the reduction ratio of the dimension of the weight matrix.

[0057] Perform a channel multiplication operation between the channel weight vector and the original sliding patch to obtain the feature map after channel domain filtering, thus completing the filtering of significant features in the channel domain.

[0058]

[0059] The process of maintaining channels is as follows: the feature map after filtering each channel domain is passed through a max pooling layer and a fully connected layer in sequence to generate spatial domain feature vectors. The attention channel processing procedure is as follows: the feature map after filtering each channel domain is sequentially passed through a max pooling layer and two fully connected layers to generate a weight vector. ;

[0060] Max pooling layer and It was set to 3 and 2, where and The kernel size and stride of the max pooling layers are set to 1, respectively. Then, two fully connected layers are used to represent the weight vectors of the local patches, with the feature vector dimensions set to 64 and 1, respectively.

[0061] Feature integration first compresses the weighted feature maps of each spatial domain through global mean pooling, and then concatenates all feature vectors to obtain a global feature vector;

[0062] In training the facial expression classification model, five-fold cross-validation was used on the controlled environment dataset, while the recognition accuracy was verified on the outdoor environment dataset using a predetermined training and test set.

[0063] The selection criteria for the optimal parameter combination are the highest recognition accuracy and reasonable model complexity;

[0064] Integrate all spatial domain weight feature maps: First, compress each spatial domain weight feature map into a 1024-dimensional feature vector through global mean pooling, and then concatenate all feature vectors to obtain the global feature vector.

[0065] The global feature vectors are fed into a fully connected layer (output dimension 6, corresponding to 6 basic expressions) and a classifier to train the expression classification model. During training, five-fold cross-validation is used on the controlled environment dataset (BU-3DFEP1, Multi-PIEP1) and the outdoor environment dataset (Pose-RAF-DB, Pose-AffectNet) to verify the recognition accuracy using predetermined training and test sets.

[0066] Adjust the spatial dimension and step size parameters, repeat the training process from S2 to S5, and conduct multiple experiments on a controlled environment dataset. Based on the criterion of "highest recognition accuracy and reasonable model complexity," select the optimal parameter combination. , .

[0067] In S6, the ResNet50 (SPA-SE) facial expression recognition network has 22.7M model parameters. On the Pose-RAF-DB dataset, the recognition accuracy is 86.76% when the head pose is >30° and 85.92% when the head pose is >45°. On the Pose-AffectNet dataset, the recognition accuracy is 59.86% when the head pose is >30° and 60.37% when the head pose is >45°.

[0068]

[0069] The SPA-SE module (composed of a sliding patch (SP) module, a channel domain attention (SE) module, and a spatial domain attention (SPA) module) can be embedded into ResNet18 and ResNet34 convolutional neural networks to achieve flexible adaptation between model complexity and recognition accuracy. Specifically, the ResNet18 (SPA-SE) network achieves a recognition accuracy of 77.64% on the BU-3DFEP1 dataset with 11.2M model parameters; the ResNet34 (SPA-SE) network achieves a recognition accuracy of 77.76% on the BU-3DFEP1 dataset.

[0070]

[0071] Use the optimal parameter combination ( , A ResNet50 (SPA-SE) facial expression recognition network was constructed, with a model parameter of 22.7M.

[0072] After the fixed-pose facial expression image to be recognized is preprocessed according to the S1 method, it is input into the ResNet50 (SPA-SE) expression recognition network, which sequentially completes the steps of deep feature extraction, sliding patch generation, channel domain attention filtering, spatial domain attention extraction, and feature fusion. Finally, the expression recognition result is output through the classifier.

[0073] Experiments show that the network achieves an accuracy of 86.76% when the head pose is >30° and 85.92% when the head pose is >45° on the Pose-RAF-DB dataset; and 59.86% when the head pose is >30° and 60.37% when the head pose is >45° on the Pose-AffectNet dataset, both of which outperform existing comparative methods.

[0074] Example 2:

[0075] The difference between this embodiment and Embodiment 1 is that the SPA-SE module is embedded into the ResNet18 network to construct a ResNet18 (SPA-SE) facial expression recognition network. The remaining steps are the same as in Embodiment 1. Experimental results show that this network achieves a recognition accuracy of 77.64% on the BU-3DFEP1 dataset, with 11.2M model parameters and a short training time. It is suitable for application scenarios with high model complexity and moderate accuracy requirements (such as embedded devices).

[0076] Example 3:

[0077] The difference between this embodiment and Embodiment 1 is that the SPA-SE module is embedded into the ResNet34 network to construct a ResNet34 (SPA-SE) facial expression recognition network. The remaining steps are the same as in Embodiment 1. Experimental results show that this network achieves a recognition accuracy of 77.76% on the BU-3DFEP1 dataset. The model parameters are between those of ResNet18 (SPA-SE) and ResNet50 (SPA-SE), making it suitable for deployment devices with moderate performance.

[0078] Working Principle: This invention generates local patch features on deep feature maps through a sliding patch (SP) module, simulating the human visual system's attention mechanism for key local regions, thus enhancing the ability to capture detailed facial expression features. The channel domain attention (SE) module assigns weights to each channel feature map, highlighting the importance of expression-related channels and suppressing noise from irrelevant channels. The spatial domain attention (SPA) module further optimizes the spatial dimension weights of the local patch features, accurately locating the spatial position information of key facial expression regions (such as the eyes and mouth). The synergistic effect of these three modules enables the network to adaptively focus on key features for facial expression recognition, effectively improving the robustness of facial expression recognition under complex poses. Specifically, the sliding patch (SP) module generates local regions by sliding on the feature map with a set spatial dimension and stride, providing refined feature input for subsequent attention mechanisms; the channel domain attention (SE) module generates channel weights through max pooling and fully connected layers, achieving feature filtering between channels; and the spatial domain attention (SPA) module generates spatial weights through similar pooling and fully connected operations, completing the ranking of spatial region importance. Finally, the feature maps optimized by both channel and spatial attention are integrated into a global feature vector through global mean pooling and concatenation, and then fed into the classifier to achieve accurate classification of facial expressions.

[0079] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A fixed-pose facial expression recognition method based on a sliding patch attention mechanism, characterized in that: Includes the following steps, S1: Data Preprocessing and Deep Feature Extraction: A fixed-pose facial expression dataset was collected and preprocessed. Deep convolutional feature maps were extracted using the Conv_1 to Conv_4 convolutional modules of a ResNet50 convolutional neural network. The size of these deep convolutional feature maps was normalized to [size missing]. 14×14 and 1024 represent the spatial and channel domain dimensions of the feature map, respectively; S2: Sliding Patch (SP) module generates local patches: To ensure that the sliding patch can accurately crop patch feature maps at different locations in the deep feature map, the SP model needs to satisfy the following constraints: in To obtain the final number of sliding patches in the SP model, The spatial dimension of the deep feature map. For positive integer local slices, and These represent the side length and step size of the sliding window in the deep feature map, respectively. This corresponds to the initial position of the sliding surface. For local slices along The sliding direction of the shaft, The number of sliding patches generated in the feature map, and the parameter and Meet the conditions This ensures that features between sliding patches maintain partial overlap, thus avoiding the defect of isolated features between traditional local patches. Local patch size... and sliding step size It can be set as needed; S3: Channel domain attention mechanism (SE) module processing; each sliding facet is fed into the bottleneck layer of the ResNet50 network, and the SE module is embedded between the two intermediate convolutional layers of the bottleneck layer; squeeze operation, activation operation and channel multiplication operation are performed in sequence to obtain the feature map after channel domain filtering; S4: Spatial Domain Attention Mechanism (SPA) Module Processing: The feature maps after filtering each channel domain are sent to the SPA module, which contains two branches: the maintenance channel and the attention channel. The maintenance channel generates spatial domain feature vectors, and the attention channel generates weight vectors. The spatial domain feature vectors and weight vectors are multiplied element-wise to obtain the spatial domain weighted feature map. S5: Feature Fusion and Optimal Parameter Selection: Integrate all spatial domain weighted feature maps to obtain a global feature vector, which is then fed into a fully connected layer and a classifier to train the facial expression classification model; adjust the spatial dimension parameters and step size parameters, conduct multiple experiments on a controlled environment dataset, and select the optimal parameter combination; S6: Fixed-pose facial expression recognition: A ResNet50 (SPA-SE) facial expression recognition network is constructed using the optimal parameter combination. The fixed-pose facial expression image to be recognized is processed according to the preprocessing method in step 1 and then input into the network. Deep feature extraction, sliding patch generation, channel domain attention filtering, spatial domain attention extraction, and feature fusion are completed in sequence. Finally, the facial expression recognition result is output through the classifier.

2. The fixed-pose facial expression recognition method based on a sliding patch attention mechanism according to claim 1, characterized in that: In S1, the fixed-pose facial expression dataset includes a controlled environment dataset (BU-3DFEP1, Multi-PIEP1) and an outdoor environment dataset (Pose-RAF-DB, Pose-AffectNet), covering extreme observation angles of 0°, 15°, 30°, 45°, 60°, 75°, and 90°, as well as six basic facial expressions: anger, disgust, happiness, fear, sadness, and surprise. Some datasets contain facial occlusion samples. The ResNet50 network is pre-trained on the ImageNet dataset to increase the representativeness of the network parameters and avoid overfitting due to insufficient data.

3. The fixed-pose facial expression recognition method based on a sliding patch attention mechanism according to claim 1, characterized in that: In S2, the dimensions of all eligible patches are then calculated using a formula. and step length The optimal parameter combination is , At this time, the ResNet50 (SPA-SE) network has a recognition accuracy of 78.01% on the BU-3DFEP1 dataset and 86.77% on the Multi-PIEP1 dataset.

4. The fixed-pose facial expression recognition method based on a sliding patch attention mechanism according to claim 1, characterized in that: In S3, the squeezing operation compresses the features of each channel into a scalar through global mean pooling. The squeezing operation formula is as follows: ,in and These represent the spatial feature dimensions of the input sliding patch in the bottleneck layer. This represents the output feature map; The activation operation is performed using a global average pooling layer and two fully connected layers. The activation operation formula is as follows: ,in and This represents the weight matrix in two fully connected layers; Represents a linear activation function; This represents the Sigmoid activation function; This indicates the reduction ratio of the dimension of the weight matrix; The output weight feature map This can be expressed as: ,in Indicates the salient features of the output channel domain. Represents a scalar parameter matrix and sliding face Perform a channel multiplication operation.

5. The fixed-pose facial expression recognition method based on a sliding patch attention mechanism according to claim 1, characterized in that: In S3, the bottleneck layer of the ResNet50 network contains four convolutional layers. The number of channels in the input and output convolutional layers is four times that of the intermediate layers. The number of channels in the input and output convolutional layers is 1024, and the number of channels in the two intermediate convolutional layers is 256. The SE module is embedded between the two intermediate convolutional layers. In the activation operation, the first fully connected layer compresses the channel feature vector to 256 dimensions (reduction ratio r=4), and the second fully connected layer restores it to 1024 dimensions.

6. The fixed-pose facial expression recognition method based on a sliding patch attention mechanism according to claim 1, characterized in that: The process of maintaining the channel is as follows: the feature map after filtering each channel domain is sequentially passed through a max pooling layer and a fully connected layer to generate a spatial domain feature vector. The processing procedure for the attention channels is as follows: the feature maps after filtering each channel domain are sequentially passed through a max pooling layer and two fully connected layers to generate a weight vector. ; The maximum pooling layer and It was set to 3 and 2, where and The kernel size and stride of the max pooling layers are set to 1, respectively. Then, two fully connected layers are used to represent the weight vectors of the local patches, with the feature vector dimensions set to 64 and 1, respectively.

7. The fixed-pose facial expression recognition method based on a sliding patch attention mechanism according to claim 1, characterized in that: The feature integration first compresses the weighted feature map of each spatial domain through global mean pooling, and then concatenates all feature vectors to obtain a global feature vector; In the training of the facial expression classification model, five-fold cross-validation is used on the controlled environment dataset, and the recognition accuracy is verified on the outdoor environment dataset using a predetermined training and test set. The selection criteria for the optimal parameter combination are the highest recognition accuracy and reasonable model complexity.

8. The fixed-pose facial expression recognition method based on a sliding patch attention mechanism according to claim 1, characterized in that: In S6, the model parameters of the ResNet50 (SPA-SE) facial expression recognition network are 22.7M; on the Pose-RAF-DB dataset, the recognition accuracy is 86.76% when the head pose is >30° and 85.92% when the head pose is >45°; on the Pose-AffectNet dataset, the recognition accuracy is 59.86% when the head pose is >30° and 60.37% when the head pose is >45°.

9. A fixed-pose facial expression recognition method based on a sliding patch attention mechanism according to claim 1, characterized in that: The SPA-SE module (composed of a sliding patch (SP) module, a channel domain attention (SE) module, and a spatial domain attention (SPA) module) can be embedded into ResNet18 and ResNet34 convolutional neural networks to achieve flexible adaptation between model complexity and recognition accuracy. Specifically, the ResNet18 (SPA-SE) network achieves a recognition accuracy of 77.64% on the BU-3DFEP1 dataset with 11.2M model parameters; the ResNet34 (SPA-SE) network achieves a recognition accuracy of 77.76% on the BU-3DFEP1 dataset.

Images