A face expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention

Abstract

The application discloses a face expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention, comprising the following steps: S1, acquiring an input face image and performing pretreatment; S2, using an adaptive super-resolution perception module to perform detail recovery or fast standardization processing on the face image; S3, using a Res2Net backbone module to encode the features of the standardized face image through a multi-scale residual structure; S4, using a multi-dimensional interactive excitation attention module to perform feature enhancement and calibration on a basic expression feature map; S5, using a full connection layer and a Softmax function to classify the enhanced features, obtaining a prediction probability distribution, and selecting an expression category with the maximum prediction probability as a final recognition result; and S6, when the model is trained, using the adaptive super-resolution perception module to perform super-resolution processing on training samples, and generating mixed enhanced samples and mixed soft labels through a super-resolution image enhancement mechanism, so as to update model parameters based on a weighted cross-entropy loss function.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, computer vision and affective computing, and specifically to a facial expression recognition method based on adaptive super-resolution enhancement and multidimensional interactive attention. Background Technology

[0002] Facial expression recognition is a crucial research task in affective computing and computer vision. Its goal is to determine the emotion category of a target based on facial texture, local deformation, regional structure, and overall appearance features in facial images or videos. With the development of convolutional neural networks and deep learning technologies, existing deep model-based facial expression recognition methods have achieved good recognition results on standard datasets such as FERPlus and RAF-DB. However, in real-world complex application scenarios, especially under conditions of long-distance shooting, multiple targets in the same scene, low-resolution imaging, target motion, and significant lighting changes, existing technologies still struggle to reliably extract effective expression features. Specifically, the following technical problems exist:

[0003] Existing technologies struggle to balance the ability to restore low-resolution facial details with recognition efficiency. In practical applications, the imaging quality and resolution of facial images are typically affected by factors such as shooting distance, camera resolution, target motion, occlusion, and scene lighting. Facial regions in distant or multi-target scenes often appear as small targets with low resolution, leading to the loss of fine-grained cues closely related to expression recognition, such as frown lines, changes in the corners of the mouth, the degree of eye closure, and subtle facial muscle deformations. Traditional image magnification methods such as bilinear interpolation and bicubic interpolation have the advantages of fast computation and simple implementation, but they essentially perform pixel-level smooth magnification and are unable to recover high-frequency texture and structural details that are discriminative for facial expression recognition. While generative adversarial networks or complex deep super-resolution methods can achieve good visual reconstruction results, their computational complexity is high. Performing complex super-resolution reconstruction uniformly on all input faces will significantly increase the inference burden, making them unsuitable for real-time recognition or multi-target facial expression recognition scenarios. Therefore, existing methods lack a super-resolution perception mechanism that can adaptively select the processing path based on the input face scale and achieve a balance between reconstruction quality and computational efficiency.

[0004] Existing training sample augmentation methods struggle to adequately improve model generalization capabilities in multi-scale and low-quality face scenarios. Facial expression datasets typically suffer from limited sample size, imbalanced class distribution, and insufficient minority class expression samples. Furthermore, the quality, resolution distribution, and acquisition conditions of training set images often differ from multi-scale inputs in real-world applications. Conventional data augmentation methods primarily expand samples through flipping, cropping, rotation, color perturbation, random occlusion, or ordinary linear mixing. These augmentation processes mostly operate directly on the original image space, failing to fully incorporate the spectral features and structural details resulting from super-resolution processing of low-resolution faces. Due to the lack of effective modeling of the continuity between super-resolution samples and their categories during training, models tend to become dependent on specific training samples and fixed resolution distributions. Consequently, their generalization ability and robustness remain insufficient when facing low-quality faces, faces across resolutions, small target faces, and similar expression categories. Therefore, existing technologies require a training augmentation mechanism that integrates super-resolution processing into the training process and enhances model generalization capabilities through mixed augmented samples and soft label constraints.

[0005] Existing attention mechanisms do not fully utilize the cross-dimensional interactions of facial expression features. Facial expression features include not only semantic responses in the channel dimension but also spatial structural information in the height and width dimensions. The correlation between these different dimensions collectively determines the model's ability to express fine-grained expression differences. Existing channel attention or spatial attention methods typically focus on weight allocation in a single dimension or sequentially weight features of channels and spatial regions, making it difficult to model the interactions between different dimensional combinations such as channel-width, channel-height, and height-width in parallel. For weak expressions, low signal-to-noise ratio images, similar expression categories, and low-resolution small face scenes, existing attention mechanisms struggle to adequately enhance the fine-grained differences in key regions such as the eyes, eyebrows, and mouth, easily leading to expression category confusion. Therefore, existing technologies require an attention mechanism capable of simultaneously performing multi-dimensional feature interactions and channel activation calibration to enhance the discriminative power of the backbone network's output features.

[0006] Existing low-resolution micro-expression recognition methods typically focus on performing unified super-resolution reconstruction on low-resolution video sequences and combining artificial spatiotemporal texture features with traditional classifiers to complete the recognition. They lack adaptive path selection mechanisms for facial inputs at different scales and do not integrate super-resolution sample fusion enhancement with the training of deep expression recognition models. While other multi-dimensional feature processing methods can be used for image classification tasks, they are mainly geared towards general target classification or specific target image classification, and do not construct a training-inference co-optimization process specifically for the low-resolution, multi-scale inputs and fine-grained expression differences encountered in facial expression recognition.

[0007] In summary, existing technologies for facial expression recognition still suffer from several challenges, including difficulty in balancing low-resolution facial detail restoration and processing efficiency, insufficient adaptation of training sample enhancement to real multi-scale inputs, and inadequate cross-dimensional expression feature modeling capabilities. Therefore, it is necessary to propose a facial expression recognition method that can adaptively select super-resolution processing paths for multi-scale facial images, generate hybrid enhanced samples and hybrid soft labels using super-resolution samples during the training phase, and perform multi-dimensional interaction and channel excitation calibration during the feature extraction phase. Summary of the Invention

[0008] The purpose of this application is to provide a facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention, the specific technical solution of which is as follows:

[0009] A facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention includes: S1, acquiring and preprocessing an input facial image; S2, using an adaptive super-resolution perception module to perform detail restoration or fast standardization on the preprocessed facial image in S1, ultimately outputting a standardized facial image; S3, using a Res2Net backbone module to encode the local texture, facial region structure, and multi-scale visual information of the standardized facial image in S2 through a multi-scale residual structure, obtaining a basic expression feature map; S4, using a multi-dimensional interactive attention module to enhance the facial expression recognition in S3. S1. Enhanced features are obtained by performing feature enhancement and calibration on the basic facial expression feature map. S2. The enhanced features in S3 are classified using a fully connected layer and a Softmax function to obtain the predicted probability distribution of each facial expression category. The facial expression category with the highest predicted probability is selected as the final recognition result, and the corresponding confidence score is output. S3. When training the model built based on the methods in S1-S5, an adaptive super-resolution perception module is used to perform super-resolution processing on the training samples. A hybrid enhanced sample and hybrid soft label are generated through a super-resolution image enhancement mechanism, and the model parameters are updated based on the weighted cross-entropy loss function.

[0010] During preprocessing in S1, when the input source is a video stream or a complex multi-target scene, the face bounding box is first obtained through the face detection model, then a single face image is obtained by cropping based on the bounding box, and finally, pixel reading, size acquisition and normalization preparation are performed.

[0011] When performing detailed restoration or rapid standardization in S2, the following are included:

[0012] S2.1, Let the input face image be... The scale metric of the input face image is The preset scale threshold is Scale awareness is performed on the input face image, and the scale measurement value is... With threshold Compare;

[0013] S2.2, when When the input face image is determined to be a low-resolution small face, it enters the convolutional super-resolution reconstruction path to perform convolutional super-resolution reconstruction.

[0014] S2.3, when When the input face image is determined to be a large or high-resolution face, it enters the fast bilinear interpolation path for size adjustment.

[0015] S2.2 includes the following when performing convolutional super-resolution reconstruction:

[0016] S2.21. Use 5×5 convolution to extract features from the input face image and map the input image to a high-dimensional feature space;

[0017] S2.22. Use 1×1 convolution to compress the feature channels to reduce computation and retain effective feature representation;

[0018] S2.23, Adopt A 3×3 convolutional layer is used for non-linear mapping to enhance the texture and structural representation in low-resolution face images;

[0019] S2.24. Use 1×1 convolution to expand channels and restore high-dimensional feature representation;

[0020] S2.25. Upsampling is performed using 9×9 deconvolution to generate a reconstructed high-resolution face image. The reconstruction process is expressed as:

[0021] ,

[0022] ,

[0023] in, This represents the feature extraction convolution operation. This indicates a channel compression convolution operation. This represents a non-linear mapping convolution operation. This indicates a channel-dilation convolution operation. This indicates the deconvolution upsampling operation. Indicates the corresponding convolution kernel parameters. This indicates the number of nonlinear mapping layers.

[0024] In S2.3, size adjustment is performed directly using bilinear interpolation to obtain the interpolated image. , expressed as:

[0025] ,

[0026] ,

[0027] in, Indicates the input image. This represents the interpolation weights based on pixel distance.

[0028] In S4, feature enhancement and calibration of the basic facial feature map includes:

[0029] S4.1. Suppose the feature map output by a certain convolutional layer of the backbone network is... ,in Indicates the number of feature channels. Indicates the feature map height. Indicates the width of the feature map;

[0030] S4.2 Perform channel-width branching, specifically including:

[0031] S4.21, For feature maps Perform a dimension permutation, swapping the channel and height dimensions of the input feature map, enabling the module to perform feature interaction under a channel-width dimension combination, resulting in... , expressed as:

[0032] ;

[0033] S4.22, regarding S4.21 Max pooling and average pooling are performed under the channel-width dimension combination, and the pooling results are concatenated before passing through a convolutional layer and a sigmoid activation function to generate attention weights. , expressed as:

[0034] ,

[0035] in, This indicates max pooling. Indicates average pooling. This indicates a splicing operation. This represents the convolution operation. This represents the Sigmoid activation function;

[0036] S4.23, Weight attention and After multiplication, the original feature map dimensions are restored by inverse dimension permutation.

[0037] S4.3 Perform channel-height branching, specifically including:

[0038] S4.31, For feature maps Perform a dimension permutation, swapping the channel and width dimensions of the input feature map, enabling the module to perform feature interaction under a channel-height dimension combination, resulting in... , expressed as:

[0039] ;

[0040] S4.32, regarding S4.31 Max pooling and average pooling are performed under the channel-height dimension combination, and the pooling results are concatenated before being passed through convolutional layers and a sigmoid activation function to generate attention weights. , expressed as:

[0041] ,

[0042] in, This indicates max pooling. Indicates average pooling. This indicates a splicing operation. This represents the convolution operation. This represents the Sigmoid activation function;

[0043] S4.33, Adjust attention weights and After multiplication, the original feature map dimensions are restored by inverse dimension permutation.

[0044] S4.4 Perform height-width branching, specifically including:

[0045] S4.41, Represent the input feature map under the height-width dimension combination as follows: ,right Max pooling and average pooling are performed in the height-width combination, and the pooling results are concatenated before being passed through a convolutional layer and a sigmoid activation function to generate attention weights. , expressed as:

[0046] ,

[0047] S4.42, Adjust attention weights and Multiply;

[0048] S4.5. Perform equal-weighted averaging fusion of the features output from the channel-width branch in S4.2, the channel-height branch in S4.3, and the height-width branch in S4.4 to obtain a multi-dimensional interactive feature map. , expressed as:

[0049] ;

[0050] S4.6 Perform channel excitation calibration, specifically including:

[0051] S4.61, regarding the multidimensional interactive feature map in S4.5 Perform global average pooling to obtain a global descriptor for each channel. , expressed as:

[0052] ,

[0053] ;

[0054] S4.62, Channel description vector Input a two-layer fully connected network and generate channel weights. , expressed as:

[0055] ,

[0056] in, For a dimension reduction matrix, For an upgraded matrix, Represents the ReLU activation function. This represents the Sigmoid activation function;

[0057] S4.7 Utilizing Channel Weights For the input feature map By performing channel-by-channel weighting, the final enhanced features are obtained. , expressed as:

[0058] ,

[0059] .

[0060] Super-resolution processing in S6 includes:

[0061] S6.1. Use face image data to train the adaptive super-resolution perception module for super-resolution reconstruction, so that it has the ability to restore low-resolution face details;

[0062] S6.2. Use the adaptive super-resolution perception module as the training front end to process the facial expression recognition training samples;

[0063] S6.3. Generate hybrid enhanced samples and hybrid soft labels through a super-resolution image enhancement mechanism;

[0064] S6.4 Input the hybrid augmented samples into the facial expression recognition model, which consists of a Res2Net backbone network, a multidimensional interactive stimulation attention module, and a classifier. Train the model and update the parameters using the weighted cross-entropy loss function.

[0065] When generating hybrid augmented samples and hybrid soft labels in S6.3, the following are included:

[0066] S6.31, For any two training samples and ,in and This represents two input face images. and The corresponding emoji category label is represented by a mixture coefficient randomly sampled from the Beta distribution. ,in Based on this mixing coefficient, a mixed enhancement sample is generated. , expressed as:

[0067] ;

[0068] S6.32, Based on the mixing coefficient Generate hybrid soft tags , expressed as:

[0069] ;

[0070] S6.33, Mix and enhance samples Inputting a facial expression recognition model, the model's predicted probability distribution is obtained through forward propagation. ;

[0071] S6.34 Constructing a weighted cross-entropy loss function using hybrid soft labels , expressed as:

[0072] ,

[0073] in, and These represent the numbers A and B in the th... Tag values ​​on emojis The model represents the first Predicted probability of facial expressions.

[0074] The beneficial effect of this application lies in the design of an adaptive super-resolution perception module that dynamically selects different resolution optimization paths based on the scale features of the input face image, rather than uniformly performing complex super-resolution reconstruction on all input images. For low-resolution, small-target faces, the module employs a convolutional super-resolution reconstruction path, recovering texture details and structural information in the face image through operations such as feature extraction, channel compression, nonlinear mapping, channel dilation, and deconvolution upsampling. For larger or higher-resolution faces, the module uses a fast bilinear interpolation path to complete size standardization processing, avoiding unnecessary complex reconstruction calculations. Through this mechanism, the processing method can be adaptively selected for face images of different scales and qualities, improving the quality of effective features of low-resolution faces while reducing redundant computational overhead, thereby improving the expression recognition performance under conditions of long distance, small targets, and low-quality faces. A super-resolution image enhancement mechanism is designed, introducing the adaptive super-resolution perception module into the model training stage. This mechanism first utilizes the ASRP module to perform super-resolution processing on the training samples, obtaining training samples with richer texture and structural information. Then, it linearly mixes two training samples based on randomly sampled mixing coefficients to generate hybrid enhanced samples, and generates corresponding hybrid soft labels according to the same mixing coefficients. Finally, it constructs a weighted cross-entropy loss function using the hybrid enhanced samples and hybrid soft labels to supervise the training of the facial expression recognition model. Compared to conventional data augmentation methods that only perform flipping, cropping, color perturbation, or ordinary linear mixing on the original image, this mechanism further utilizes the detailed features and continuous relationships between samples after super-resolution processing, enabling the model to learn transitional features between samples of different resolutions, different facial expression categories, and different image qualities during the training phase. Through this mechanism, this invention can alleviate the problems of insufficient generalization ability caused by the limited number of training samples, imbalanced category distribution, and inconsistency between the resolution distribution of the training set and the real application scenario. A multi-dimensional interactive stimulation attention module is designed and embedded into the feature extraction process of the facial expression recognition backbone network. This module constructs channel-width, channel-height, and height-width branches for the feature maps output by the backbone network, respectively, to model feature dependencies in different dimensional combinations in parallel. Then, the interactive features output by the three branches are fused to obtain a multi-dimensional interactive feature representation. Furthermore, channel calibration weights are generated through global average pooling and a two-layer fully connected network, and these weights are used to apply incentive-based weighting to key facial expression feature channels. Through this mechanism, this invention can simultaneously capture spatial dependencies, channel dependencies, and cross-dimensional dependencies, enhancing the expressive power of key facial expression regions such as eyes, eyebrows, and mouth, as well as related channel features, while suppressing redundant or interfering features. This improves the model's ability to distinguish similar expressions, weak expressions, and low-resolution small facial expressions.

[0075] This application clearly distinguishes between the training and testing / inference phases in its overall process, and assigns ASRP, SRIA, and MDIEA to different technical stages. Specifically, the ASRP module is used for multi-scale face input processing in both the training and testing / inference phases, dynamically selecting either convolutional super-resolution reconstruction or fast bilinear interpolation paths based on the face scale. The SRIA mechanism is only used in the training phase, generating hybrid enhanced samples and hybrid soft labels from the ASRP-processed super-resolution samples to optimize model parameters; it does not participate in the testing / inference phase. The MDIEA module is embedded in the feature extraction process of the Res2Net backbone network, used for multi-dimensional interactive modeling and channel activation calibration of backbone features. Instead of simply combining low-resolution super-resolution reconstruction methods with multi-dimensional feature processing methods, this application forms a continuous technical chain of "multi-scale input processing—training sample enhancement—deep feature calibration—expression classification output." This process can simultaneously improve problems such as the lack of detail in low-resolution faces, insufficient generalization ability of training samples, and inadequate expression of cross-dimensional facial features, thereby improving the recognition stability and robustness of the facial expression recognition model in complex scenarios. Attached Figure Description

[0076] Figure 1 This is a schematic diagram of the overall method flow of this application;

[0077] Figure 2 This is a schematic diagram of the adaptive super-resolution sensing module structure in this application;

[0078] Figure 3 This is a schematic diagram of the super-resolution image enhancement mechanism in this application;

[0079] Figure 4 This is a schematic diagram of the multidimensional interactive stimulation attention module structure in this application. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0081] like Figure 1 As shown, a facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention includes:

[0082] S1. Acquire the input face image and perform preprocessing. Specifically, during preprocessing, when the input source is a video stream or a complex multi-object scene, the face bounding box is first obtained through a face detection model, then a single face image is obtained by cropping based on the bounding box, and finally, pixel reading, size acquisition, and normalization preparation are performed.

[0083] like Figure 2 As shown, S2 uses an adaptive super-resolution perception module to perform detail restoration or fast standardization on the preprocessed face image in S1, ultimately outputting a standardized face image. Specifically, the detail restoration or fast standardization process includes:

[0084] S2.1, Let the input face image be... The scale metric of the input face image is The preset scale threshold is Scale awareness is performed on the input face image, and the scale measurement value is... With threshold Compare;

[0085] S2.2, when When the input face image is determined to be a low-resolution small face, it enters the convolutional super-resolution reconstruction path to perform convolutional super-resolution reconstruction.

[0086] S2.3, when When the input face image is determined to be a large or high-resolution face, it enters the fast bilinear interpolation path for size adjustment.

[0087] S2.2 includes the following when performing convolutional super-resolution reconstruction:

[0088] S2.21. Use 5×5 convolution to extract features from the input face image and map the input image to a high-dimensional feature space;

[0089] S2.22. Use 1×1 convolution to compress the feature channels to reduce computation and retain effective feature representation;

[0090] S2.23, Adopt A 3×3 convolutional layer is used for non-linear mapping to enhance the texture and structural representation in low-resolution face images;

[0091] S2.24. Use 1×1 convolution to expand channels and restore high-dimensional feature representation;

[0092] S2.25. Upsampling is performed using 9×9 deconvolution to generate a reconstructed high-resolution face image. The reconstruction process is expressed as:

[0093] ,

[0094] ,

[0095] in, This represents the feature extraction convolution operation. This indicates a channel compression convolution operation. This represents a non-linear mapping convolution operation. This indicates a channel-dilation convolution operation. This indicates the deconvolution upsampling operation. Indicates the corresponding convolution kernel parameters. This indicates the number of nonlinear mapping layers.

[0096] In S2.3, size adjustment is performed directly using bilinear interpolation to obtain the interpolated image. , expressed as:

[0097] ,

[0098] ,

[0099] in, Indicates the input image. This represents the interpolation weights based on pixel distance.

[0100] In applications, the final output of the adaptive super-resolution sensing module can be expressed as:

[0101] ,

[0102] ,

[0103] Through the above processing, the adaptive super-resolution perception module can perform detail restoration for low-resolution small faces and perform fast standardization for high-quality faces, thereby improving the quality of effective features of low-resolution faces while avoiding redundant computation. The module outputs a standardized face image. As input to the subsequent backbone network.

[0104] S3. The Res2Net backbone module is used to encode the local texture, facial region structure and multi-scale visual information of the standardized face image in S2 through a multi-scale residual structure to obtain the basic expression feature map.

[0105] like Figure 4 As shown, in step S4, a multi-dimensional interactive attention module is used to enhance and calibrate the basic facial expression feature map in step S3, resulting in enhanced features. Specifically, the feature enhancement and calibration of the basic facial expression feature map includes:

[0106] S4.1. Suppose the feature map output by a certain convolutional layer of the backbone network is... ,in Indicates the number of feature channels. Indicates the feature map height. Indicates the width of the feature map;

[0107] S4.2 Perform channel-width branching, specifically including:

[0108] S4.21, For feature maps Perform a dimension permutation, swapping the channel and height dimensions of the input feature map, enabling the module to perform feature interaction under a channel-width dimension combination, resulting in... , expressed as:

[0109] ;

[0110] S4.22, regarding S4.21 Max pooling and average pooling are performed under the channel-width dimension combination, and the pooling results are concatenated before passing through a convolutional layer and a sigmoid activation function to generate attention weights. , expressed as:

[0111] ,

[0112] in, This indicates max pooling. Indicates average pooling. This indicates a splicing operation. This represents the convolution operation. This represents the Sigmoid activation function;

[0113] S4.23, Weight attention and After multiplication, the original feature map dimensions are restored by inverse dimension permutation.

[0114] S4.3 Perform channel-height branching, specifically including:

[0115] S4.31, For feature maps Perform a dimension permutation, swapping the channel and width dimensions of the input feature map, enabling the module to perform feature interaction under a channel-height dimension combination, resulting in... , expressed as:

[0116] ;

[0117] S4.32, regarding S4.31 Max pooling and average pooling are performed under the channel-height dimension combination, and the pooling results are concatenated before being passed through convolutional layers and a sigmoid activation function to generate attention weights. , expressed as:

[0118] ,

[0119] in, This indicates max pooling. Indicates average pooling. This indicates a splicing operation. This represents the convolution operation. This represents the Sigmoid activation function;

[0120] S4.33, Adjust attention weights and After multiplication, the original feature map dimensions are restored by inverse dimension permutation.

[0121] S4.4 Perform height-width branching, specifically including:

[0122] S4.41, Represent the input feature map under the height-width dimension combination as follows: ,right Max pooling and average pooling are performed in the height-width combination, and the pooling results are concatenated before being passed through a convolutional layer and a sigmoid activation function to generate attention weights. , expressed as:

[0123] ,

[0124] S4.42, Adjust attention weights and Multiply;

[0125] S4.5. Perform equal-weighted averaging fusion of the features output from the channel-width branch in S4.2, the channel-height branch in S4.3, and the height-width branch in S4.4 to obtain a multi-dimensional interactive feature map. , expressed as:

[0126] ;

[0127] S4.6 Perform channel excitation calibration, specifically including:

[0128] S4.61, regarding the multidimensional interactive feature map in S4.5 Perform global average pooling to obtain a global descriptor for each channel. , expressed as:

[0129] ,

[0130] ;

[0131] S4.62, Channel description vector Input a two-layer fully connected network and generate channel weights. , expressed as:

[0132] ,

[0133] in, For a dimension reduction matrix, For an upgraded matrix, Represents the ReLU activation function. This represents the Sigmoid activation function;

[0134] S4.7 Utilizing Channel Weights For the input feature map By performing channel-by-channel weighting, the final enhanced features are obtained. , expressed as:

[0135] ,

[0136] .

[0137] S5. The enhanced features in S4 are classified using a fully connected layer and a Softmax function to obtain the predicted probability distribution of each expression category. The expression category with the highest predicted probability is selected as the final recognition result, and the corresponding confidence level is output.

[0138] like Figure 3 As shown in Figure S6, when training the model built based on methods S1-S5, an adaptive super-resolution perception module is used to perform super-resolution processing on the training samples. A super-resolution image enhancement mechanism is then used to generate hybrid enhanced samples and hybrid soft labels, which are updated based on the weighted cross-entropy loss function. Specifically, the super-resolution processing includes:

[0139] S6.1. Use face image data to train the adaptive super-resolution perception module for super-resolution reconstruction, so that it has the ability to restore low-resolution face details;

[0140] S6.2. Use the adaptive super-resolution perception module as the training front end to process the facial expression recognition training samples;

[0141] S6.3. Generate hybrid enhanced samples and hybrid soft labels through a super-resolution image enhancement mechanism;

[0142] S6.4 Input the hybrid augmented samples into the facial expression recognition model, which consists of a Res2Net backbone network, a multidimensional interactive stimulation attention module, and a classifier. Train the model and update the parameters using the weighted cross-entropy loss function.

[0143] When generating hybrid augmented samples and hybrid soft labels in S6.3, the following are included:

[0144] S6.31, For any two training samples and ,in and This represents two input face images. and The corresponding emoji category label is represented by a mixture coefficient randomly sampled from the Beta distribution. ,in Based on this mixing coefficient, a mixed enhancement sample is generated. , expressed as:

[0145] ;

[0146] S6.32, Based on the mixing coefficient Generate hybrid soft tags , expressed as:

[0147] ;

[0148] S6.33, Mix and enhance samples Inputting a facial expression recognition model, the model's predicted probability distribution is obtained through forward propagation. ;

[0149] S6.34 Constructing a weighted cross-entropy loss function using hybrid soft labels , expressed as:

[0150] ,

[0151] in, and These represent the numbers A and B in the th... Tag values ​​on emojis The model represents the first Predicted probability of facial expressions.

[0152] To make this application easier to understand, the following examples illustrate it further.

[0153] Example 1: Training of facial expression recognition models based on ASRP, SRIA and MDIEA.

[0154] The adaptive super-resolution perception module (ASRP) is pre-trained using a face image dataset, and the expression recognition model is trained and tested using a face expression dataset. Training samples for the ASRP module can be generated from high-resolution face images through downsampling to produce low-resolution input. The corresponding original high-resolution image or a standardized high-quality face image is used as the supervision target. The ASRP module is trained using a pixel-level mean square error loss function, enabling it to recover detailed information from low-resolution face images. The convolutional super-resolution reconstruction path in the ASRP module can be set as follows: number of feature extraction channels... Number of compressed channels nonlinear mapping layer number The deconvolution upsampling kernel size is Scale threshold The scale threshold can be set according to the specific application scenario, face detection bounding box size, input image resolution, and device computing power. In a specific experimental setup, the scale threshold can be... Set the threshold to 60 pixels when inputting face scale. When the input image enters the convolutional super-resolution reconstruction path; when At this time, the input image enters the fast bilinear interpolation path. During the training phase of the facial expression recognition model, Res2Net is used as the backbone network to extract basic facial expression features, and a multidimensional interactive stimulation attention module is embedded in the feature extraction process of the backbone network. The MDIEA module performs feature interaction modeling on the feature map output by the backbone network in three branches: channel-width, channel-height, and height-width, and further enhances key facial expression features through channel stimulation calibration.

[0155] Simultaneously, a super-resolution image enhancement mechanism is enabled during the training phase. For any two samples in the training set... and First, super-resolution samples are obtained by processing them separately using the ASRP module, and then the mixture is processed according to the random sampling mixing coefficients. Generate hybrid enhanced samples and mixed soft labels The hybrid augmented samples are then input into an expression recognition model consisting of a Res2Net backbone network, an MDIEA module, and a classifier for training. The Beta distribution parameter in the SRIA mechanism can be set to... The mixed trigger probability can be set to 0.5; the channel downsampling rate in the MDIEA module. It can be set to 16.

[0156] The standardized face image is then input into the Res2Net backbone network to extract basic expression features; then, a multi-dimensional interaction-stimulated attention module is used for multi-dimensional feature interaction and channel calibration; finally, the expression category and its confidence score are output through a classifier and a Softmax function. No super-resolution image enhancement mechanism is executed during the testing / inference phase. In a specific training setup, the AdamW optimizer can be used, with an initial learning rate of 0.001, a batch size of 128, 50 training epochs, and a weight decay of 1e−4, employing a cosine learning rate scheduling strategy or other equivalent learning rate scheduling strategies. The above parameters are only one specific implementation method and do not constitute a limitation on the scope of protection of this invention.

[0157] Example 2:

[0158] During the inference phase, the input is a face image or video frame to be identified. If the input is a video frame or a multi-object scene image, the face detection module can first obtain one or more face bounding boxes, and then crop a single face image based on the face bounding boxes. For each cropped face image, its scale index is calculated. Then input it into the ASRP module.

[0159] when At that time, the ASRP module calls the convolutional super-resolution reconstruction path to perform feature extraction, channel compression, nonlinear mapping, channel dilation, and deconvolution upsampling on the low-resolution small target face, generating a reconstructed high-resolution face image; when At this time, the ASRP module calls the fast bilinear interpolation path to perform size normalization on larger or higher resolution face images. The outputs of both paths are then normalized in size and pixels to obtain a standardized face image. .

[0160] Subsequently, standardized facial images The input is a Res2Net backbone network embedded with the MDIEA module. The backbone network extracts basic facial expression features, and the MDIEA module performs multidimensional interaction modeling and channel excitation calibration on the feature maps. The enhanced facial expression features are then input into a classifier and a Softmax function, ultimately outputting the facial expression category and its confidence level.

[0161] In this embodiment, the SRIA mechanism is only used for sample augmentation and loss calculation during the training phase and does not participate in the inference phase, thus not adding any additional sample mixing calculations during the inference phase. The ASRP module adaptively selects either the convolutional reconstruction path or the fast interpolation path based on the face scale during the inference phase; the MDIEA module is embedded in the backbone network as a feature enhancement component to improve the expressive and discriminative power of classification features.

[0162] The module combination effect is verified as follows:

[0163] A facial expression recognition model without ASRP, SRIA, and MDIEA was used as the baseline model. ASRP, SRIA, and MDIEA modules were gradually added for ablation comparison. In one validation method, the ablation configuration shown in Table 1 illustrates the function of each module. Table 1 shows that after adding ASRP, SRIA, and MDIEA to the baseline model, the recognition accuracy on the FERPlus dataset increased from 88.12% to 90.71%, and the recognition accuracy on the RAF-DB dataset increased from 87.66% to 89.21%.

[0164] Table 1

[0165]

[0166] The ASRP module primarily improves the quality of face images under low-resolution or multi-scale input conditions; the SRIA mechanism mainly expands the training sample distribution and enhances the model's generalization ability; and the MDIEA module mainly enhances cross-dimensional expression feature representation and fine-grained feature discrimination capabilities. ASRP, SRIA, and MDIEA have clear functional divisions in the training and inference phases: SRIA is mainly used for sample augmentation during training, ASRP is used for multi-scale input processing during both training and inference, and MDIEA is used for feature enhancement during the backbone network feature extraction phase. Their synergistic effect effectively improves the recognition stability of the face expression recognition model under complex conditions such as low-resolution input, imbalanced sample distribution, and differentiation of similar expressions.

Claims

1. A facial expression recognition method based on adaptive super-resolution enhancement and multidimensional interactive attention, characterized in that, include: S1. Obtain the input face image and perform preprocessing; S2. An adaptive super-resolution perception module is used to perform detail restoration or rapid standardization on the preprocessed face image in S1, and finally outputs a standardized face image. S3. Using the Res2Net backbone module, the local texture, facial region structure and multi-scale visual information of the standardized face image in S2 are encoded through a multi-scale residual structure to obtain the basic expression feature map. S4. Use a multi-dimensional interactive attention module to enhance and calibrate the basic facial expression feature map in S3 to obtain enhanced features; S5. The enhanced features in S4 are classified using a fully connected layer and a Softmax function to obtain the predicted probability distribution of each expression category. The expression category with the highest predicted probability is selected as the final recognition result, and the corresponding confidence level is output. S6. When training the model constructed based on the methods in S1-S5, an adaptive super-resolution perception module is used to perform super-resolution processing on the training samples, and a hybrid enhanced sample and hybrid soft label are generated through a super-resolution image enhancement mechanism to update the model parameters based on the weighted cross-entropy loss function.

2. The facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention as described in claim 1, characterized in that, During preprocessing in S1, when the input source is a video stream or a complex multi-target scene, the face bounding box is first obtained through a face detection model, then a single face image is obtained by cropping based on the bounding box, and finally, pixel reading, size acquisition and normalization preparation are performed.

3. The facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention as described in claim 2, characterized in that, The detailed restoration or rapid normalization process in S2 includes: S2.1, Let the input face image be... The scale metric of the input face image is The preset scale threshold is Scale awareness is performed on the input face image, and the scale measurement value is... With threshold Compare; S2.2, when When the input face image is determined to be a low-resolution small face, it enters the convolutional super-resolution reconstruction path to perform convolutional super-resolution reconstruction. S2.3, when When the input face image is determined to be a large or high-resolution face, it enters the fast bilinear interpolation path for size adjustment.

4. The facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention as described in claim 3, characterized in that, The convolutional super-resolution reconstruction in S2.2 includes: S2.

21. Use 5×5 convolution to extract features from the input face image and map the input image to a high-dimensional feature space; S2.

22. Use 1×1 convolution to compress the feature channels to reduce computation and retain effective feature representation; S2.23, Adopt A 3×3 convolutional layer is used for non-linear mapping to enhance the texture and structural representation in low-resolution face images; S2.

24. Use 1×1 convolution to expand channels and restore high-dimensional feature representation; S2.

25. Upsampling is performed using 9×9 deconvolution to generate a reconstructed high-resolution face image. The reconstruction process is expressed as: ， ， in, This represents the feature extraction convolution operation. This indicates a channel compression convolution operation. This represents a non-linear mapping convolution operation. This indicates a channel-dilation convolution operation. This indicates the deconvolution upsampling operation. Indicates the corresponding convolution kernel parameters. This indicates the number of nonlinear mapping layers.

5. The facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention as described in claim 4, characterized in that, In step S2.3, the size adjustment is performed directly using bilinear interpolation to obtain the interpolated image. , expressed as: ， ， in, Indicates the input image. This represents the interpolation weights based on pixel distance.

6. The facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention as described in claim 5, characterized in that, The feature enhancement and calibration of the basic facial expression feature map in S4 includes: S4.

1. Suppose the feature map output by a certain convolutional layer of the backbone network is... ,in Indicates the number of feature channels. Indicates the feature map height. Indicates the width of the feature map; S4.2 Perform channel-width branching, specifically including: S4.21, For feature maps Perform a dimension permutation, swapping the channel and height dimensions of the input feature map, enabling the module to perform feature interaction under a channel-width dimension combination, resulting in... , expressed as: ； S4.22, regarding S4.21 Max pooling and average pooling are performed under the channel-width dimension combination, and the pooling results are concatenated before passing through a convolutional layer and a sigmoid activation function to generate attention weights. , expressed as: ， in, This indicates max pooling. Indicates average pooling. This indicates a splicing operation. This represents the convolution operation. This represents the Sigmoid activation function; S4.23, Adjust attention weights and After multiplication, the original feature map dimension is restored by inverse dimension permutation; S4.3 Perform channel-height branching, specifically including: S4.31, For feature maps Perform a dimension permutation, swapping the channel and width dimensions of the input feature map, enabling the module to perform feature interaction under a channel-height dimension combination, resulting in... , expressed as: ； S4.32, regarding S4.31 Max pooling and average pooling are performed under the channel-height dimension combination, and the pooling results are concatenated before being passed through convolutional layers and a sigmoid activation function to generate attention weights. , expressed as: ， in, This indicates max pooling. Indicates average pooling. This indicates a splicing operation. This represents the convolution operation. This represents the Sigmoid activation function; S4.33, Adjust attention weights and After multiplication, the original feature map dimension is restored by inverse dimension permutation; S4.4 Perform height-width branching, specifically including: S4.41, Represent the input feature map under the height-width dimension combination as follows: ,right Max pooling and average pooling are performed in the height-width combination, and the pooling results are concatenated before being passed through a convolutional layer and a sigmoid activation function to generate attention weights. , expressed as: ， S4.42, Weighting attention and Multiply; S4.

5. The features output from the channel-width branch in S4.2, the channel-height branch in S4.3, and the height-width branch in S4.4 are fused by equal weighting to obtain a multi-dimensional interactive feature map. , expressed as: ； S4.6 Perform channel excitation calibration, specifically including: S4.61, regarding the multidimensional interaction feature map in S4.5 Perform global average pooling to obtain a global descriptor for each channel. , expressed as: ， ； S4.62, Channel description vector Input a two-layer fully connected network and generate channel weights. , expressed as: ， in, For a dimension reduction matrix, For an upgraded matrix, Represents the ReLU activation function. This represents the Sigmoid activation function; S4.7 Utilizing Channel Weights For the input feature map By performing channel-by-channel weighting, the final enhanced features are obtained. , expressed as: ， 。 7. The facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention as described in claim 6, characterized in that, The super-resolution processing in S6 includes: S6.

1. Use face image data to train the adaptive super-resolution perception module for super-resolution reconstruction, so that it has the ability to restore low-resolution face details; S6.

2. Use the adaptive super-resolution perception module as the training front end to process the facial expression recognition training samples; S6.

3. Generate hybrid enhanced samples and hybrid soft labels through a super-resolution image enhancement mechanism; S6.4 Input the hybrid augmented samples into the facial expression recognition model, which consists of a Res2Net backbone network, a multidimensional interactive stimulation attention module, and a classifier. Train the model and update the parameters using the weighted cross-entropy loss function.

8. The facial expression recognition method based on adaptive super-resolution enhancement and multi-dimensional interactive attention as described in claim 7, characterized in that, The generation of hybrid enhanced samples and hybrid soft labels in S6.3 includes: S6.31, For any two training samples and ,in and This represents two input face images. and The corresponding emoji category label is represented by a mixture coefficient randomly sampled from the Beta distribution. ,in Based on this mixing coefficient, a mixed enhancement sample is generated. , expressed as: ； S6.32, Based on the mixing coefficient Generate hybrid soft tags , expressed as: ； S6.33, Mix and enhance samples Inputting a facial expression recognition model, the model's predicted probability distribution is obtained through forward propagation. ; S6.34 Constructing a weighted cross-entropy loss function using hybrid soft labels , expressed as: ， in, and These represent the numbers A and B in the th... Tag values ​​on emojis The model represents the first Predicted probability of facial expressions.

Images