A face expression recognition method based on global and local feature refinement

Abstract

The application discloses a face expression recognition method based on global and local feature fine-tuning, and belongs to the technical field of computer vision and pattern recognition. The method pre-processes an input face image, extracts image features and key point features respectively, performs global dependence modeling and local fine-grained deformation enhancement on the key point features, and performs cross-modal fusion on the image features; expression discrimination features are obtained through compact representation mapping and multi-scale coding, and are input into a classification module to output an expression category. During training, a label smoothing and category weighting joint supervision strategy is adopted to improve the learning ability of minority class samples. The application can improve the distinguishing ability of similar expression categories, and reduce the parameter quantity and the calculation complexity.

Description

Technical Field

[0001] This invention relates to the field of computer vision and pattern recognition technology, and in particular to a deep learning-based facial expression recognition method, which can be applied to scenarios such as affective computing, human-computer interaction, educational analysis, medical and health monitoring, public safety and driving status recognition. Background Technology

[0002] Facial expression recognition refers to the technology of automatically analyzing and determining an individual's emotional state from static or dynamic facial images. It is one of the important research directions in the intersection of affective computing and computer vision. Because facial expressions can reflect an individual's emotional changes, interest levels, confusion levels, and stress levels, facial expression recognition has high application value in fields such as healthcare, public safety, educational assessment, social robotics, and driver fatigue monitoring.

[0003] As facial expression recognition technology moves from controlled experimental environments to complex real-world scenarios, its performance is affected by a variety of factors. Existing methods typically face the following problems in practical applications: First, limited training data size and unstable annotation quality easily lead to model overfitting. Second, factors irrelevant to facial expression, such as changes in lighting, head pose, resolution degradation, partial occlusion, and individual identity differences, can interfere with the extraction of effective facial expression features, thereby reducing the model's generalization ability. Furthermore, some expression categories exhibit high similarity, while significant differences exist within the same category. Additionally, the uneven distribution of categories in the dataset makes it difficult for minority class samples to learn sufficiently, further impacting recognition stability.

[0004] Early facial expression recognition methods often employed handcrafted features such as local binary patterns, Gabor features, and histograms of oriented gradients, combined with shallow classification models like support vector machines. While these methods performed well in controlled scenarios, their robustness to complex backgrounds, occlusion, pose variations, and lighting disturbances was weak. With the development of deep learning, convolutional neural networks (CNNs) enhanced their ability to model local texture information through end-to-end feature learning, improving the overall performance of facial expression recognition. However, CNNs still have relatively limited ability to model long-range dependencies and fall short in characterizing the relationship between global semantics and key local regions.

[0005] In recent years, Transformer structures based on self-attention mechanisms have been introduced into facial expression recognition tasks. These methods can model long-range dependencies between different regions in the feature representation space, thus facilitating the simultaneous capture of global semantics and local discriminative cues. Building upon this, facial expression recognition methods that integrate appearance information and structural priors, combined with multi-scale interaction mechanisms, have further improved the ability to focus on expression regions in complex scenes. Methods represented by the pyramid cross-fusion Transformer network, by introducing keypoint information and cross-stream interaction mechanisms, have improved expression recognition accuracy to a certain extent.

[0006] However, existing methods still have the following shortcomings: First, to achieve multi-branch feature fusion and cross-stream interaction, it is often necessary to introduce additional branches and more complex interaction modules, which increases the number of parameters and computational complexity, making them unsuitable for deployment in resource-constrained scenarios. Second, although existing methods can model certain global relationships, they are still insufficient in depicting fine-grained deformations in key local areas such as eyebrows, eyes, and corners of the mouth, and there is still room for improvement in their ability to distinguish between similar expression categories. Third, under data conditions with uneven class distribution, minority class samples account for a low proportion in training, which can easily lead to insufficient learning of weaker classes by the model, thereby affecting macro-average performance and class balance.

[0007] Therefore, it is necessary to propose a facial expression recognition method that can simultaneously take into account global dependency modeling, local fine-grained deformation enhancement, compact feature representation, and class imbalance learning, so as to improve the model's recognition performance, computational efficiency, and practical deployment feasibility. Summary of the Invention

[0008] The problem to be solved

[0009] The purpose of this invention is to provide a facial expression recognition method based on global and local feature refinement, addressing the problems of existing facial expression recognition methods in real-world scenarios, such as large inter-class similarity, large intra-class differences, scale sensitivity, and unbalanced class distribution. The existing POSTER framework improves expression recognition performance to some extent through bi-branch cross-modal fusion of image features and keypoint features. However, the keypoint branch still has room for improvement in its ability to express local fine-grained deformations, and the high fusion embedding dimension introduces additional parameters and computational burden. This invention improves the network from two directions—keypoint enhancement and dimensionality reduction—without altering the core of the POSTER bi-branch cross-modal fusion.

[0010] Technical solution

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] A facial expression recognition method based on global and local feature refinement includes the following steps:

[0013] S1. Obtain the face image to be identified and preprocess the face image; the preprocessing includes face detection, face alignment, cropping, size normalization and pixel normalization to obtain a standardized input image.

[0014] S2. Input the preprocessed face image into the image backbone network and the face landmark detector respectively to extract the image token sequence and the landmark token sequence; wherein, the image backbone network is used to extract the overall appearance features, and the landmark detector is used to extract the structural prior features related to expression.

[0015] S3. Embedding alignment and dimensionality compression are performed on the image label sequence and keypoint label sequence. Specifically, the image labels are linearly projected from 1024 dimensions to 384 dimensions, and the keypoint labels are linearly projected from 512 dimensions to 384 dimensions to complete cross-modal embedding alignment and reduce the parameter scale of subsequent attention and feedforward networks.

[0016] S4. Introduce global and local feature refinement modules in the keypoint branch to jointly refine the projected keypoint labels. The global branch models the global dependencies between labels through multi-head self-attention, while the local branch models the local adjacency relationships and fine-grained non-rigid deformations through channel-wise one-dimensional convolution. The global refinement increment and the local refinement increment are then fused with the original keypoint labels in a residual manner to obtain the refined keypoint labels.

[0017] S5. Construct sequence-level representations for the image label sequence and the refined keypoint label sequence respectively; perform mean pooling on the two labels to obtain the corresponding Mean-CLS labels, and concatenate the Mean-CLS labels with their respective label sequences to form the image enhancement sequence and the keypoint enhancement sequence.

[0018] S6. Input the image enhancement sequence and keypoint enhancement sequence into the cross-modal fusion coding module, and update the image branch and keypoint branch representations respectively through bidirectional cross attention; then construct a three-scale pyramid branch on the feature dimension to perform multi-scale modeling and cross-scale fusion of the fused features.

[0019] S7. Input the large-scale branch output representation into the task head, and add a channel recalibration module before classification to enhance the effective channel response and suppress redundant features; then output the expression category classification output value through a linear classification layer.

[0020] S8. During the training phase, the label smooth cross-entropy loss and the class-weighted cross-entropy loss are combined as the final optimization objective to enhance the learning ability of minority class and long-tail class samples.

[0021] Beneficial effects

[0022] Compared with the prior art, the present invention has at least the following beneficial effects:

[0023] (1) By simultaneously modeling global dependencies and local neighborhood deformations in the key point labeling space, the ability to perceive subtle facial expression changes in key areas such as eyebrows, eyes, and corners of the mouth is enhanced.

[0024] (2) By using unified embedding dimension compression and multi-scale pyramid coding, the number of parameters and computational complexity are reduced while maintaining recognition performance;

[0025] (3) By combining label smoothing with category-weighted joint loss, the recognition stability of minority class samples and long-tail classes is improved;

[0026] (4) Experimental results show that the method of the present invention has achieved better recognition results on RAF-DB, FERPlus and AffectNet datasets, and has shown good generalization ability and deployment feasibility. Attached Figure Description

[0027] Figure 1 This is a flowchart of a facial expression recognition method based on global and local feature refinement according to the present invention.

[0028] Figure 2 This is a schematic diagram of the structure of the global and local feature refinement module of the present invention.

[0029] Figure 3 This is a schematic diagram of the overall network structure of the present invention. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited to the following embodiments.

[0031] Example 1: Overall Method Flow

[0032] like Figure 1 As shown, this embodiment provides a facial expression recognition method based on global and local feature refinement. The method includes steps such as input image preprocessing, dual-branch feature extraction, embedding alignment and dimensionality compression, keypoint feature refinement, cross-modal fusion, multi-scale encoding, classification output, and training optimization. Specifically, a global and local feature refinement module is set in the keypoint branch, and the computational complexity is reduced and the learning ability of minority class samples is enhanced through embedding dimensionality compression and class imbalance optimization strategies. In this embodiment, the method optimizes the facial expression recognition process by reducing dimensionality and enhancing keypoint features while maintaining the core structure of dual-stream encoding and cross-modal fusion.

[0033] Specifically, the system first acquires the face image to be identified and performs preprocessing such as face detection, alignment, cropping, scaling, and normalization. Then, the preprocessed face image is input into the image backbone network and the keypoint detector, respectively, to extract the image label sequence and the keypoint label sequence. Then, embedding alignment and dimensionality compression are performed on the two feature streams, and only the keypoint labels are introduced into the global and local feature refinement modules for discrimination enhancement. After that, multi-scale modeling is achieved through cross-modal label concatenation and cross-fusion pyramid Transformer encoding. Finally, the expression category is output through the classification head, and label smoothing and class-weighted joint loss are used for optimization during the training phase.

[0034] Example 2: Dual-branch feature encoding and embedding alignment

[0035] In this embodiment, the input image is fed into an image backbone network and a face keypoint detector, respectively, to obtain image feature sequences and keypoint feature sequences. The image backbone network is used to extract the overall appearance representation of the face, while the keypoint detector is used to extract prior structural representations related to facial expressions.

[0036] To reduce model complexity and unify the embedding spaces of the two modalities, in this embodiment, image labels are linearly mapped from 1024 dimensions to 384 dimensions, and keypoint labels are linearly mapped from 512 dimensions to 384 dimensions. After dimensionality compression, the two feature streams undergo subsequent refinement and fusion operations in a unified embedding space, thereby reducing the parameter size of the attention module and the feedforward network.

[0037] In a preferred embodiment, the image backbone network may employ an IR50 or other convolutional neural network backbone to extract appearance representations of the input face image; the keypoint detector may employ MobileFaceNet or other lightweight detection networks capable of outputting a predetermined number of facial keypoints to extract expression-related structural prior representations. In one embodiment, during the training phase, the image backbone network participates in parameter updates, while the keypoint detector remains frozen to stabilize keypoint representations.

[0038] Example 3: Sequence-level representation construction and cross-modal fusion input

[0039] In this embodiment, the image branch output is assumed to be... The key branch output is ,in, Indicates batch size. This indicates the number of tags in each branch. This represents a unified embedding dimension. To construct sequence-level representations, mean pooling is performed on both image tags and keypoint tags to obtain the corresponding Mean-CLS tags:

[0040] , Equation (1)

[0041] And further form an enhanced sequence:

[0042] , Equation (2)

[0043] Here, "[;]" indicates concatenation along the label dimension. In a preferred implementation, positional encoding is applied only to the image branches. After the above processing, the two enhanced sequences are used for subsequent cross-modal fusion, and the length of the fused sequence after concatenation is 100.

[0044] Example 4: Global and Local Feature Refinement Module

[0045] like Figure 2 As shown, in this embodiment, the global and local feature refinement modules only apply to the keypoint branch and are set after dimensionality projection and before cross-modal fusion. Given a keypoint label sequence, layer normalization is first performed to obtain intermediate features:

[0046] Equation (3)

[0047] Then, the global refinement branch and the local refinement branch are calculated separately. The global branch models global dependencies between labels using multi-head self-attention; the local branch models local adjacency relationships and fine-grained non-rigid deformations using channel-wise one-dimensional convolution, and its output is:

[0048] Equation (4)

[0049] Wherein, DWConv1d represents a one-dimensional convolution along the label dimension, the kernel size is preferably 3, the padding is preferably 1, and the number of groups is equal to the channel dimension.

[0050] Furthermore, the global branch output and the local branch output are fused with the original features using a residual method to obtain:

[0051] Equation (5)

[0052] Then update via feedforward network:

[0053] Equation (6)

[0054] Among them, the relevant parameter is the learnable scaling coefficient, which is preferably set to 0 during initialization to reduce the perturbation of the pre-training distribution in the early stage of training. The global and local feature refinement module enhances the discriminativeness and stability of key point representation by jointly modeling global dependencies and local fine-grained changes.

[0055] Example 5: Cross-fusion pyramid Transformer coding

[0056] like Figure 3 As shown, in this embodiment, the image enhancement sequence and the keypoint enhancement sequence are input into the cross-modal fusion coding module. Bidirectional cross-attention is used to update the image branch and keypoint branch representations respectively, thereby establishing the correlation between the two modalities and enhancing complementary information interaction. In one embodiment, bidirectional cross-attention is used to update the image branch representation and the keypoint branch representation respectively. For the landmark→image update method, the keypoint branch enhancement sequence is used as the query, and the image branch enhancement sequence is linearly mapped to obtain the key and value, thereby updating the image branch representation; for the image→landmark update method, the image branch enhancement sequence is used as the query, and the keypoint branch enhancement sequence is linearly mapped to obtain the key and value, thereby updating the keypoint branch representation. In a preferred implementation, the query directly retrieves the other modality marker, and the key and value are linearly mapped and then output as a projection to reduce the number of parameters and maintain the effectiveness of cross-modal interaction.

[0057] To mitigate scale sensitivity, a three-scale feature branch is constructed by performing two downsampling operations along the feature dimension while keeping the number of labels constant. Preferably, the embedding dimensions for the large-scale, medium-scale, and small-scale branches are 384, 192, and 96, respectively. After feature modeling is performed on each of the three-scale branches, the medium-scale and small-scale features are upsampled and backpropagated to the large-scale branch to complete cross-scale information fusion. Alternatively, to further mitigate scale sensitivity while keeping the number of labels constant, a three-scale representation can be constructed using one-dimensional convolution along the feature dimension. In the original POSTER settings, the three-scale dimensions can be: In the lightweight design of this invention, the three-scale dimensions are preferably... Features from the mesoscale and small-scale branches are upsampled after their respective modeling is completed and then backpropagated to the large-scale branch to achieve cross-scale fusion.

[0058] Finally, the image classification labels of the large-scale branches are taken as the input head of the global fusion representation task, and a channel recalibration module is added before classification to enhance the effective channel response and suppress redundant features. Subsequently, the expression category classification output value is output through a linear classification layer.

[0059] Example 6: Handling Imbalanced Training Objectives and Classes

[0060] In this embodiment, the facial expression recognition task is modeled as a multi-class classification problem. For the first... For each sample, the model output is processed using softmax to obtain the predicted probability. To improve training stability and long-tail class recognition ability, a joint optimization of label-smoothed cross-entropy and class-weighted cross-entropy is employed. The label-smoothed soft target distribution is defined as:

[0061] Equation (7)

[0062] in For the smoothing coefficient, take... The corresponding label smoothing cross-entropy loss is:

[0063] Equation (8)

[0064] The category-weighted cross-entropy loss is:

[0065] Equation (9)

[0066] in For the first Class weight, This represents the sample weights obtained by indexing according to the true class. The final training objective is:

[0067] Equation (10)

[0068] The final training objective is obtained by weighted summation of label-smooth cross-entropy loss and class-weighted cross-entropy loss, with the loss weight coefficients... The optimal choice is 2. Different datasets can adopt different imbalanced processing strategies based on the degree of long tail. For example, RAF-DB and FERPlus use loss layer reweighting, while AffectNet can combine data layer resampling.

[0069] In one implementation, the class weights in the class-weighted cross-entropy are determined based on the number of training samples for each class; preferably, the class weights can be calculated using the effective number method. To reduce interference with the supervision of the head classes, weighting can be enabled only for the tail classes, while the weight of the head classes is fixed at 1. In another implementation, for datasets with a higher degree of long-tail distribution of classes, data layer resampling can be introduced during the training data loading phase. Specifically, sampling weights can be assigned to each training sample according to the sample frequency of the class to which the sample belongs, and sampling probabilities can be constructed based on the sampling weights, thereby improving the class composition of the mini-batch.

[0070] Example 7: Training Configuration and Dataset Setup

[0071] To verify the effectiveness of the method of this invention, the RAF-DB, FERPlus, and AffectNet datasets can be used for training and evaluation. The RAF-DB dataset has approximately 12,271 training images and approximately 3,068 test images, corresponding to 7 basic expression categories; the FERPlus dataset has approximately 28,709 training images and approximately 3,589 test images, corresponding to 8 expression categories; the AffectNet subset can be constructed using stratified sampling, with 24,855 training images and 3,500 test images for the 7-category setting, and 27,299 training images and 4,000 test images for the 8-category setting.

[0072] In this embodiment, the experiment can be implemented under the PyTorch framework, and training and evaluation can be completed on a single NVIDIA RTX 4090 GPU. The runtime environment can be Python 3.8 and CUDA 11.3. All input face images are uniformly scaled to 224×224, and random horizontal flipping and random erasing are used as data augmentation methods.

[0073] On the RAF-DB dataset, the preferred batch size is 144, the preferred number of training epochs is 200, the preferred initial learning rate is 3.5 × 10⁻⁵, and the preferred weight decay coefficient is 1 × 10⁻⁵. -4 An exponentially decaying learning rate scheduling strategy is adopted; on the FERPlus dataset, the batch size is preferably set to 144, the number of training epochs is preferably 200, and the initial learning rate is preferably 4×10⁻⁶. -5 The training employs warmup and cosine learning rate scheduling. On the AffectNet dataset, the preferred batch size is 144, the preferred number of training epochs is 200, and the preferred initial learning rate is 3.5 × 10⁻⁶. -5 Furthermore, an imbalanced sampling strategy is introduced during the data loading phase to improve the class distribution of mini-batch.

[0074] In this embodiment, different datasets can be processed using different imbalanced methods depending on the degree of long tail. RAF-DB and FERPlus can employ a loss layer reweighting strategy, while AffectNet can employ a loss layer reweighting and / or a data layer resampling strategy. The above configuration is only a preferred embodiment, and those skilled in the art can make adjustments without departing from the essence of the invention.

[0075] Table 1 Dataset and Training Configuration of Embodiments of the Invention

[0076]

[0077] Example 8: Results of Effect Verification

[0078] In this embodiment, the method of the present invention is compared with the baseline method and the POSTER method. Experimental results show that the method of the present invention achieves superior recognition performance on multiple standard datasets.

[0079] For example, on the RAF-DB dataset, the accuracy of the method of this invention reached 92.28%, and the average class accuracy reached 86.61%; compared with the baseline of 91.00% and 84.64%, it improved by 1.28 percentage points and 1.97 percentage points, respectively; compared with the POSTER dataset of 92.05% and 86.03%, it improved by 0.23 percentage points and 0.58 percentage points, respectively.

[0080] On the FERPlus dataset, the accuracy of the method of this invention reached 91.73%, which is an improvement of 0.82 percentage points compared with the baseline of 90.91% and an improvement of 0.11 percentage points compared with the POSTER dataset of 91.62%.

[0081] On the AffectNet dataset, with a 7-class setting, the accuracy of the proposed method reaches 64.89%, an improvement of 2.40 percentage points compared to the baseline of 62.49%; with an 8-class setting, the accuracy reaches 60.23%, an improvement of 1.53 percentage points compared to the baseline of 58.70%. These results demonstrate that the proposed method exhibits good generalization ability under complex scenarios and class imbalance conditions.

[0082] To further verify the lightweight effect of the method of this invention, the parameter count, computational cost, and inference speed of the method of this invention were compared with those of POSTER. The results show that the parameter count of the method of this invention is 55.97M, which is about 22.0% lower than POSTER's 71.8M; the computational cost is 14.12 G FLOPs, which is about 10.1% lower than POSTER's 15.7 G FLOPs; the inference time is 22.1 ms, which is better than POSTER's 25.7 ms; and the inference speed is 44.7 fps, which is higher than POSTER's 38.9 fps. These results demonstrate that the present invention can reduce model complexity and improve deployment feasibility while maintaining high recognition performance.

[0083] Table 2 Comparison of recognition results between the method of this invention, the baseline method, and POSTER.

[0084]

[0085] Example 9: Ablation and Visualization Explanation

[0086] To further verify the contributions of each technical feature of this invention, step-by-step analysis was conducted on the RAF-DB and AffectNet datasets. Experimental results show that after introducing class-weighted loss on top of the baseline, the recognition performance on both RAF-DB and AffectNet is improved, indicating that the class-weighted strategy can alleviate the training bias caused by class imbalance. After further introducing global and local feature refinement modules, the model's recognition performance on both datasets continues to improve, indicating that the combination of global dependency modeling and local neighborhood deformation modeling can enhance the expression discrimination ability. After further introducing embedding dimension compression, the model reduces feature redundancy and model complexity while maintaining high recognition performance. After combining class weighting, global and local feature refinement, and 384-dimensional compact embedding, the final model achieves stable gains on both RAF-DB and AffectNet relative to the baseline, indicating that there is synergistic gain among the components of this invention.

[0087] Furthermore, visualization analysis of the model's attention response reveals that the high-response regions of the method in this invention are more concentrated in key areas related to facial expression changes, including the eyebrow and eye regions, the corners of the mouth, the cheeks, and the forehead, while the response to the background and non-key areas is relatively weak. Compared to POSTER, the high-response regions of the method in this invention are more concentrated and more stably cover the expression discrimination-related regions. This visualization result is consistent with the changing trends of the class-by-class recognition rate and macro-average index, qualitatively demonstrating the enhancement effect of the global and local feature refinement modules on key expression regions.

[0088] Example 10: Preferred Implementation

[0089] In a preferred embodiment, the image backbone network is used to extract the appearance representation of the input face image, and the key point detector is used to extract the corresponding facial key point structure representation; the image backbone network may adopt a convolutional neural network backbone or a Transformer backbone, and the key point detector may adopt a lightweight detection network capable of outputting a predetermined number of facial key points.

[0090] In this preferred embodiment, the initial dimension of the image markers output by the image branch is 1024, and the initial dimension of the keypoint markers output by the keypoint branch is 512. They are compressed to a unified 384-dimensional embedding space by linear mapping. Subsequently, only the keypoint marker sequence is refined with global and local features, and then input together with the image marker sequence into the cross-modal fusion coding module.

[0091] In this preferred embodiment, the local branches in the global and local feature refinement modules adopt channel-wise one-dimensional convolution set along the label dimension, the convolution kernel size is preferably 3, and the padding is preferably 1; the embedding dimensions of the multi-scale pyramid coding branches are preferably 384, 192, and 96, respectively.

[0092] During the training phase, it is preferable to use a joint loss function consisting of label smooth cross-entropy loss and class-weighted cross-entropy loss, and select loss layer reweighting and / or data layer resampling strategies according to the class distribution of different datasets.

Claims

1. A facial expression recognition method based on global and local feature refinement, characterized in that, Includes the following steps: S1. Obtain the face image to be identified, and perform face detection, alignment, cropping, scaling and normalization preprocessing on the face image; S2. Input the preprocessed face image into the image backbone network and the key point detector respectively, and extract the image label sequence and the key point label sequence; S3. Embedding alignment and dimensionality compression are performed on the image label sequence and key point label sequence to map the two feature paths to a unified embedding space. S4. Perform global and local feature refinement only on the key point label sequence. The global and local feature refinement includes: modeling the global dependency relationship between labels through a multi-head self-attention mechanism, and modeling the local neighborhood relationship and fine-grained non-rigid deformation through a channel-wise one-dimensional convolution set along the label dimension. S5. Input the refined keypoint label sequence and image label sequence into the cross-modal fusion coding module, and perform intermodal feature interaction through cross attention; S6. Construct a multi-scale pyramid coding branch for the cross-modal fusion features and fuse features at different scales; S7. Take the fused classification label as the global representation, and output the expression category result through the classification layer.

2. The facial expression recognition method according to 1, characterized in that, In step S3, the image markers are mapped from 1024 dimensions to 384 dimensions, and the keypoint markers are mapped from 512 dimensions to 384 dimensions.

3. The facial expression recognition method according to 1, characterized in that, In step S4, the global and local feature refinement module includes layer normalization, multi-head self-attention branch, channel-wise one-dimensional convolution branch, residual fusion unit, and feedforward network unit.

4. The facial expression recognition method according to 3, characterized in that, The channel-wise one-dimensional convolution branch adopts channel-wise one-dimensional convolution set along the label dimension, with a kernel size of 3, padding of 1, and the number of groups equal to the channel dimension.

5. The facial expression recognition method according to 3, characterized in that, The multi-head self-attention branch output and the channel-wise one-dimensional convolution branch output are fused with the original keypoint label features through residual methods, and a learnable scaling coefficient is set to weight each branch output.

6. The facial expression recognition method according to 1, characterized in that, In step S6, a three-scale feature branch is constructed, with embedding dimensions of 384, 192, and 96 for large, medium, and small scales, respectively.

7. The facial expression recognition method according to 1, characterized in that, During model training, a joint loss function is constructed using label-smoothed cross-entropy loss and class-weighted cross-entropy loss. Class imbalance is addressed by weighting the loss layer and / or resampling the data layer based on the distribution of samples of each class in the training data.

8. A facial expression recognition device, characterized in that, It includes a preprocessing module, a bi-branch feature extraction module, an embedding alignment and compression module, a global and local feature refinement module, a cross-modal fusion module, a multi-scale coding module, and a classification output module; wherein, the classification output module is used to perform channel recalibration on the fused representation and output the expression category result; the modules work together to implement the facial expression recognition method according to any one of 1 to 7.

9. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when executed by the processor, the computer program implements the facial expression recognition method according to any one of 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the facial expression recognition method according to any one of 1 to 7.

Images