Face emotion feature extraction method based on healthy-affected side contrast channel attention

Abstract

The application discloses a kind of healthy-patient side contrast passage attention human face emotion feature extraction method, belong to medical image processing and affective computing cross field.Aiming at the facial asymmetry distortion caused by cerebral apoplexy facial paralysis, the input feature map is split into left and right half faces along the facial sagittal axis and mirror alignment in the application, and the bilateral difference matrix coding space asymmetry is calculated;Determine the orientation of healthy and sick sides dynamically through a lightweight branch network, use healthy side features to compensate for sick side features;Embed an adaptive multi-scale fusion module, and assign different receptive field branch weights based on input feature responses;Use teacher network to perform spatial attention and soft label joint distillation on student network;During training, combine H-B grading simulation data augmentation, phased dynamic resampling and class weighted loss.The application explicitly uses facial paralysis pathological priori, effectively suppresses the interference of asymmetric distortion, has higher recognition accuracy than existing models, and has small parameter quantity and fast inference speed, suitable for deployment on rehabilitation edge devices.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of medical image processing, computer vision and emotion computing. Specifically, it relates to a method and recognition system for extracting emotion features based on the attention mechanism of the healthy-affected side contrast channel for the scene of local facial distortion caused by central facial paralysis after stroke. Background Technology

[0002] Stroke is one of the leading causes of death and long-term disability in adults worldwide. According to the World Health Organization, approximately 15 million new strokes occur annually, with more than two-thirds of survivors experiencing varying degrees of neurological impairment. Central facial paralysis is one of the most common sequelae of stroke, characterized by flaccid paralysis of the lower facial muscles (orbicularis oris, buccinator, levator anguli oris, etc.) on the contralateral side of the lesion, resulting in a crooked mouth, shallowing or disappearance of the nasolabial fold, and facial asymmetry. Facial paralysis not only severely impairs a patient's appearance and social function but also induces negative emotions such as anxiety and depression due to impaired facial expression, creating a vicious cycle of physiological and psychological distress that greatly hinders the rehabilitation process. Therefore, real-time and accurate identification of the patient's emotional state during rehabilitation treatment has irreplaceable clinical value for adjusting rehabilitation strategies, evaluating treatment effectiveness, and preventing psychological crises.

[0003] Facial Expression Recognition (FER) is a core task in affective computing. In recent years, deep learning-based methods have made significant progress in general scenarios. For example, networks such as ResNet, VGG, and Transformer have achieved high classification accuracy on benchmark datasets such as FER2013, AffectNet, and RAF-DB. Lightweight models such as MobileNet and ShuffleNet have also been successfully deployed on mobile devices, enabling real-time inference.

[0004] However, directly applying existing FER technology to stroke patients with facial paralysis faces three major technical bottlenecks:

[0005] First, pathological distortions cause feature extraction to fail. Existing convolutional neural networks (CNNs) and visual transformers (ViTs) implicitly assume that the human face has global symmetry or isotropy. Their core operations—global average pooling (GAP) and convolution with symmetrical receptive fields—severely smooth out the asymmetrical topology between the healthy and affected sides. For example, when a patient has right-sided facial paralysis causing the right corner of the mouth to droop while the left corner is normal, global pooling mixes and averages the pixels of the left and right sides, causing key emotion discrimination features (such as the angle of the upturned corner of the mouth and the difference in the depth of the nasolabial fold) to be submerged by pathological distortions. The model cannot effectively distinguish between easily confused emotions such as "sadness," "neutrality," and even "disgust." Experiments show that on the same distorted data, the classic ResNet50 scores less than 40% for the emotions "fear" and "disgust."

[0006] Second, clinical data is scarce, and simulations are detached from pathological constraints. Facial images of real stroke patients involve highly sensitive medical privacy, and obtaining large-scale datasets with emotion-annotated and facial paralysis level-annotated faces legal and ethical barriers. Furthermore, annotation requires the participation of professional physicians, which is extremely costly. Current researchers mostly use common image augmentation techniques such as random affine transformation, random occlusion, and elastic deformation for data augmentation. However, these augmentation methods do not consider the key clinical features of central facial paralysis—preservation of upper facial (frontalis and orbicularis oculi muscles) function (innervated by bilateral cortical structures) and unilateral paralysis of the lower face (perioral region). The resulting simulation data is severely disconnected from real pathology, leading to extremely poor generalization ability of the trained model in actual rehabilitation scenarios, a phenomenon known as the "simulation-reality gap."

[0007] Third, there is a conflict between model complexity and edge deployment requirements. High-performance FER models (such as ResNet50 and ViT-Base) have a large number of parameters (>50M) and high computational cost (>10GFLOPs), making them difficult to deploy on rehabilitation edge devices (such as embedded GPUs, mobile terminals, and rehabilitation robots). On the other hand, lightweight models (such as MobileNetV3, with <5M parameters) have significantly insufficient discrimination ability on distorted faces, failing to meet clinical accuracy requirements.

[0008] Regarding existing attention mechanisms: SENet compresses channel information through global pooling, completely discarding spatial location; CBAM considers both channel and space, but its spatial attention is still generated based on global channel pooling; coordinate attention (CA) partially retains positional information, but still does not explicitly model "left-right asymmetry". None of these mechanisms have the ability to perform mirror comparison and feature compensation for facial paralysis pathology.

[0009] Therefore, there is an urgent need for an emotion recognition method that can fully utilize clinical priors, accurately capture asymmetric distortion features, and meet the requirements of lightweight deployment. Summary of the Invention

[0010] Purpose of the invention

[0011] This invention addresses the shortcomings of existing technologies by providing a method for extracting facial emotion features from a healthy-affected side contrast channel. It aims to solve the problems of decreased emotion recognition performance and difficulty in lightweight model deployment caused by facial asymmetry distortion in stroke-induced facial paralysis. This invention achieves compensation for emotion features from the healthy side to the affected side by explicitly encoding a "symmetry breaking" prior. Combined with multi-scale dynamic fusion, clinical knowledge-driven data augmentation, and heterogeneous knowledge distillation, it significantly improves the accuracy and robustness of emotion recognition in distorted faces, while maintaining model parameter and computational requirements suitable for edge deployment.

[0012] Technical solution

[0013] To achieve the above objectives, the present invention provides the following technical solution:

[0014] A method for extracting facial emotion features based on healthy-affected side contrast channel attention includes the following steps:

[0015] Step S1: Obtain the face image to be identified, and input the face image into a pre-trained deep convolutional neural network, which includes a healthy-affected side contrast channel attention module;

[0016] Step S2: Perform the following operations in the healthy-affected side contrast channel attention module:

[0017] S2.1 Spatial Splitting and Mirror Alignment: The input feature map is split into left and right half-face features along the mid-sagittal axis of the face, and one side is horizontally flipped to make the anatomical positions of the two sides correspond one-to-one.

[0018] S2.2 Difference Matrix Encoding: Calculate the feature difference matrix between the left half of the face and the flipped right half of the face. This difference matrix is ​​used to quantify the degree of spatial asymmetry between the left and right sides of the face. The calculation formula is as follows:

[0019]

[0020] S2.3 Horizontal Attention Generation: The difference matrix is ​​pooled in one dimension along the width direction, and horizontal attention weights are generated through nonlinear mapping to suppress asymmetric pathological noise;

[0021] S2.4 Healthy Side Dynamic Evaluation: The left half-face features, the mirror-aligned right half-face features, and the difference matrix are all subjected to global average pooling. The pooling results are concatenated and then sequentially input into the first fully connected layer, the ReLU activation layer, the second fully connected layer, and the Softmax layer to obtain the probability that the left side is the relatively healthy side and the probability that the right side is the relatively healthy side. Based on the two probabilities, the left half-face features and the mirror-aligned right half-face features are continuously weighted through a healthy-affected side soft routing mechanism to obtain the relatively healthy side features and the relatively affected side features.

[0022] S2.5 Feature Compensation: Based on the difference between the relative healthy side feature and the relative affected side feature, a healthy side compensation mask is generated. This mask is then used to perform soft-gated fusion of the relative healthy side feature and the relative affected side feature to generate the compensated affected side feature. The calculation formula is as follows:

[0023]

[0024]

[0025] Step S3: Input the feature map processed in step S2 into the subsequent convolutional layer and classifier, and output the emotion classification result of the face image.

[0026] Furthermore, the deep convolutional neural network also includes an adaptive multi-scale feature fusion module (SK-MSPA), which performs the following operations:

[0027] The input features are reduced in dimensionality by 1×1 convolutions and then fed in parallel into three depthwise separable convolutional branches with different dilation rates, namely: The corresponding receptive fields are 3×3, 5×5, and 7×7, and the output of each branch is denoted as... ;

[0028] Will Element-by-element addition yields the fusion feature The channel descriptor is obtained by global average pooling. Then, a selection weight vector is generated through the bottleneck fully connected layer. :

[0029]

[0030] in , Compression ratio ;

[0031] right Applying Softmax normalization yields the branch weights. And calculate the dynamic fusion output:

[0032]

[0033] Will After recovering the channels through pointwise convolution, they are connected to the original input residuals.

[0034] Furthermore, the deep convolutional neural network employs a lightweight network based on the ResNet34 backbone. The healthy-affected side contrast channel attention module is located at the Stage 2 output, used to extract left-right asymmetric features of the corners of the mouth, nasolabial folds, and lower facial contours at a higher spatial resolution, and to complete the healthy-affected side determination and affected side feature compensation. The SK-MSPA module is located after Stage 3 and before Stage 4, used for adaptive multi-scale fusion of the compensated features.

[0035] Furthermore, the training of the deep convolutional neural network employs a teacher-student knowledge distillation strategy, where the teacher network is ConvNeXt-Tiny and the student network is a lightweight ResNet34 embedded with CCA and SK-MSPA; the distillation loss includes:

[0036] Soft-label KL divergence loss:

[0037]

[0038] Spatial attention MSE loss:

[0039]

[0040] Total loss:

[0041] .

[0042] Furthermore, the method also includes a clinical knowledge-driven data augmentation step: based on the House-Brackmann facial paralysis grading standard, the original healthy face image is upsampled using bicubic interpolation, and 68 facial key points are extracted using Dlib, with the intercanthal distance between the two eyes as the basis. As a rigid normalized baseline, the target displacement of key points on the affected side is determined based on the mapping of light, moderate, and severe distortion parameters. The global dense deformation field is solved using thin plate spline interpolation (TPS).

[0043]

[0044] in The coefficients are obtained by aligning the source control points with the target control points, which are radial basis functions. .

[0045] A smooth, decaying weighting function is applied to the perioral region to eliminate artifacts.

[0046]

[0047] Furthermore, a phased dynamic resampling strategy and category-weighted cross-entropy loss are employed during the training phase to mitigate the long-tail distribution problem of sentiment categories. Category weight calculation:

[0048]

[0049] in The total number of samples in the training set. For the number of emotion categories, For the first The number of samples in the class.

[0050] Beneficial effects

[0051] Compared with the prior art, the present invention has the following significant advantages:

[0052] 1. Explicit Encoding of Pathological Asymmetry: By mirror image splitting and difference matrix calculation, the core clinical prior of "health-affected side asymmetry" in facial paralysis following stroke is structurally injected into the attention mechanism, avoiding the problem of general attention losing key spatial information due to global pooling. (Difference matrix) It can be directly visualized, providing physicians with interpretable localization of pathological areas.

[0053] 2. Dynamic identification and feature compensation of the affected and healthy sides: Without requiring external labels for the affected side, the model can adaptively determine the orientation of the healthy side (outputting the probability of the healthy side) and use the relatively complete facial expression features of the healthy side to compensate for and repair the affected side. This is the first of its kind in known literature, greatly improving the robustness of features under pathological distortions.

[0054] 3. Adaptive multi-scale receptive field: The SK-MSPA module dynamically adjusts the fusion weights based on the distortion level of the input samples. A single model can cover all levels of scenarios, from light to heavy, with only about 0.2M more parameters, which is far less than traditional multi-branch parallel structures (such as Inception).

[0055] 4. Lightweight and High-Precision: Using ResNet34 as the backbone and embedding CCA and SK-MSPA, the total number of parameters is 21.69M, with an inference latency of only 13.2ms / frame on the CPU. It achieves 70.15% accuracy on the self-built distortion FER2013 test set and 65.32% accuracy on the real facial paralysis FPED validation set, both outperforming the more parameter-intensive ResNet50 and POSTER V2.

[0056] 5. Clinical knowledge-driven simulation data construction: The data generation method based on HB grading and TPS algorithm makes the training data highly consistent with the real pathological features of "preservation of the upper face and unilateral distortion of the lower face", effectively bridging the domain gap between simulation and clinical practice and improving the model's generalization ability.

[0057] 6. Comprehensive training strategy: Combining dynamic resampling, cost-sensitive loss, label smoothing, EMA, mixed precision training, and heterogeneous knowledge distillation, the model's convergence stability and generalization ability are guaranteed under extremely imbalanced and noisy data. Attached Figure Description

[0058] Figure 1 This is a diagram of the overall network architecture of the present invention, which shows the input image, the ResNet34 backbone, the location of the embedded CCA module, the location of the SK-MSPA module, and the classification output layer.

[0059] Figure 2 This is a detailed structural diagram of the healthy-affected side contrast channel attention (CCA) module of the present invention, including spatial splitting and flipping, difference matrix calculation, horizontal attention generation, vertical attention generation, healthy side dynamic evaluation sub-network and feature compensation fusion path.

[0060] Figure 3 This is a structural block diagram of the adaptive multi-scale feature fusion module (SK-MSPA) of the present invention, which includes multi-branch depthwise separable convolution, element-wise addition, global average pooling, bottleneck full connection, softmax weight allocation and residual connection.

[0061] Figure 4 This is a comparison of the distortion simulation effects based on the House-Brackmann classification of this invention, showing examples of images of the original healthy face, mild distortion, moderate distortion, and severe distortion, as well as corresponding schematic diagrams of the displacement field of facial key points.

[0062] Figure 5 This is a heatmap of the confusion matrix of the present invention on the distorted FER2013 dataset, comparing the recall rates of the baseline model and the present invention for each category.

[0063] Figure 6 This is a visualization comparison of the feature clustering (t-SNE) of this invention and the baseline model, showing the intra-class compactness and inter-class separation of the feature space.

[0064] Figure 7 This is a schematic diagram of the thin plate spline interpolation (TPS) transformation in this invention, showing the source control point, the target control point, and the generated smooth deformation mesh.

[0065] Figure 8This is a graph of the perioral smoothing attenuation weight function in this invention. The horizontal axis represents the normalized distance from the pixel to the healthy side corner of the mouth, and the vertical axis represents the displacement weight.

[0066] Figure 9 This is a structural diagram of the healthy side dynamic evaluation subnetwork in this invention, which includes GAP, FC1, ReLU, FC2, Softmax, and healthy / affected side soft routing mechanism.

[0067] Figure 10 This is a diagram showing the learning rate scheduling curve and the training / validation loss versus accuracy curve in this invention. Detailed Implementation

[0068] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. All equivalent substitutions or improvements made based on the principles of the present invention should be included within the scope of protection of the present invention.

[0069] Example 1: Overall Network Architecture

[0070] This embodiment provides a method for extracting facial emotion features based on healthy-affected side contrast channel attention. Its overall network architecture is as follows: Figure 1 As shown. The network uses ResNet34 as its backbone, and its structural parameters are as follows:

[0071] Input layer: Receives a 224×224×3 (or 224×224×1 grayscale) image, which is then processed by 7×7 convolution (stride 2, output channels 64), batch normalization, ReLU, and 3×3 max pooling (stride 2).

[0072] Stage 1: 3 residual blocks, 64 output channels, no downsampling.

[0073] Stage 2: 4 residual blocks. The first residual block is downsampled (step size 2, output channels 128), and a CCA module is embedded after the residual blocks in Stage 2 (see...). Figure 2 ).

[0074] Stage 3: 6 residual blocks, the first residual block is downsampled (output channels 256), and an SK-MSPA module is set after all residual blocks (see...). Figure 3 ).

[0075] Stage 4: 3 residual blocks, the first residual block is downsampled (output channel 512), and the output feature size is 7×7.

[0076] Classification head: Global average pooling layer + Dropout (0.5) + fully connected layer (512→7), outputting 7 categories of emotion probabilities.

[0077] Example 2: Specific Implementation of the CCA Module

[0078] like Figure 2 As shown, the input feature map ,in For batch size, For the number of channels, This refers to the spatial dimensions.

[0079] Step ①: Split and flip: along the horizontal center line ( The face was divided into the left half. and the right half of the face .Will Horizontally flip (mirror) to obtain , making Middle position ( ) corresponds to the original right half of the face ( ).

[0080] Step ② Difference Matrix: Calculate This embodiment takes (Square difference). The shape of the difference matrix is... . Areas with larger median values ​​indicate strong asymmetry between the left and right halves of the face in that spatial location and along that channel, corresponding to the pathological area of ​​facial paralysis.

[0081] Step ③ Generation of Horizontal Attention: Along the height direction ( Adaptive average pooling is performed on the axis to obtain :

[0082]

[0083] Then, horizontal attention weights are generated through 1×1 convolution (reducing the number of channels to 1 while keeping the width and height unchanged) and sigmoid activation. Finally, it is extended to include... through a broadcast mechanism. Same space size However, it only affects the horizontal positions corresponding to the left and right halves of the face.

[0084] Step 4: Y-path (vertical attention): For the original input Average pooling is performed along the width direction to obtain :

[0085]

[0086] Vertical attention weights are generated by 1×1 convolution (output channel 1) and sigmoid activation. This weight can enhance the model's focus on the lower half of the face (a region with a high incidence of pathological conditions).

[0087] Step ⑤ Dynamic evaluation of the healthy side subnetwork: such as Figure 9 As shown, input , and The vector is obtained by global average pooling and then concatenation. Then enter the following in sequence:

[0088] Fully connected layer Output Dimension ;

[0089] ReLU activation;

[0090] Fully connected layer Output dimension 2;

[0091] Softmax: Output , ;

[0092] Healthy and Affected Side Soft Router: Output , .

[0093] Judgment rule: Based on the relative healthy side probability output by Softmax, perform soft routing for the healthy and affected sides, where... This represents the probability that the left side is the healthy side. This represents the probability that the right side is the healthy side. (Through...) and The features of the left and right halves of the face are continuously weighted to obtain the features of the healthy side. Characteristics of the affected side This avoids misallocation of healthy and affected sides due to hard threshold determination. The calculation formula is as follows:

[0094]

[0095]

[0096] Step 6: Feature Compensation: Taking the determination that the left side is the healthy side and the right side is the affected side as an example, extract features from the healthy side. Characteristics of the affected side .right Perform horizontal flip (mirror transformation) Obtain the features of the healthy side that are spatially aligned with the affected side. Then fuse them according to the following formula:

[0097]

[0098] This embodiment takes If the right side is determined to be the healthy side, then the corresponding swap is performed. If the condition is determined to be symmetrical, then no compensation or setting is performed. .

[0099] Step 7: Attention Weighting and Output: Weight the original features... Integrate with horizontal and vertical attention:

[0100]

[0101] in For element-wise multiplication, Indicates will and Broadcast to The spatial dimensions are summed. Then, the compensatory features are added:

[0102]

[0103] in This is a learnable scalar parameter, initialized to 0.5. Finally... The output of the CCA module is passed to the next layer.

[0104] Example 3: Specific Implementation of the SK-MSPA Module

[0105] like Figure 3 As shown, input features .

[0106] ①Split: First, the number of channels is halved using a 1×1 pointwise convolution, resulting in... ,in Then, three parallel depthwise separable convolutions were performed, each with a kernel size of 3×3 and dilatation rates of [missing information]. The stride is 1, and the padding is 1, 2, and 3 respectively to ensure that the output size remains unchanged. Each depthwise convolution is followed by a 1×1 pointwise convolution, and the output channels are preserved. The output has three branches. .

[0107] ②Fuse (Fusing and Global Information Embedding): Adds the outputs of the three branches element by element.

[0108]

[0109] Then, global average pooling is used to... Compressed into channel descriptor :

[0110]

[0111] Next, a selection weight vector is generated through the bottleneck fully connected layer. :

[0112]

[0113] in The compression ratio in this embodiment ; ; .

[0114] ③Select (Dynamic Selection): For Applying the Softmax function yields the normalized weights for each branch:

[0115]

[0116] The dynamic fusion output is:

[0117]

[0118] ④ Output and residual connection: Connect the output and residual. By 1×1 pointwise convolution (the output channels are restored to...) ,get Then, with the original input Add the residuals:

[0119]

[0120] This is the output of the SK-MSPA module. This module adaptively adjusts the receptive field based on the input features, even with slight distortion. Larger, severe distortion Relatively large.

[0121] Example 4: Clinical Knowledge-Driven Data Augmentation

[0122] To address the scarcity of real-world data on facial paralysis following stroke, this invention constructs a simulation training set based on the House-Brackmann (HB) classification standard. The specific steps are as follows:

[0123] Step A: Image preprocessing and key point detection

[0124] The original FER2013 dataset images are 48×48 pixel grayscale images. Due to the low resolution, Dlib's 68-point facial keypoint detection model fails. Therefore, bicubic interpolation is first used to upsample the images by a factor of 4 to 192×192 pixels. Then, Dlib's regression tree ensemble model is used to detect 68 keypoints, and the successfully detected samples are selected.

[0125] Define the coordinates of the inner canthi (medial corners of the eyes) of both eyes, and calculate their Euclidean distance as a rigid normalization reference. All subsequent displacements are divided by To eliminate the influence of individual differences and shooting distance.

[0126] Step B: Set distortion parameters according to HB classification

[0127] The clinical features of central facial paralysis are: basic symmetry of the upper face (eyebrows and eyes), and paralysis on the affected side of the lower face (perioral and mandibular regions). Therefore, we set the key points of the upper face (upper contour, eyebrows, and eyes) as rigid anchor points, maintaining their coordinates during deformation. The key points of the affected side of the lower face were parametrically displaced according to the HB classification. Table 1 shows the parameter mapping for three levels of distortion:

[0128] Table 1. Parameter mapping for third-order distortion

[0129] Mild Level II Symmetrical at rest, slightly asymmetrical in motion 0.08 0.02 none moderate Level III-IV The face is noticeably asymmetrical when at rest, with flat cheeks. 0.12 0.06 Local contrast decreased by 20% on the affected side. Severe V-VI level Almost no facial movement, severe asymmetry 0.20 0.10 The lower lip shifts 0.05D towards the healthy side.

[0130] The target displacement of the corner of the mouth on the affected side is:

[0131]

[0132] Direction: Vertically downwards ( ), horizontally towards the healthy side (if the left side is the healthy side, then the corner of the mouth on the affected side moves to the right). .

[0133] Step C: Thin Plate Spline Interpolation (TPS) Deformation

[0134] The original coordinates of 68 key points were used as source control points. The coordinates after displacement are used as the target control points. The global dense deformation field is solved using TPS. The radial basis functions of TPS are: The deformation function is expressed as:

[0135]

[0136] in By solving the system of linear equations (satisfying...) (and minimizing bending energy) yields the coefficient Then, the deformed coordinates of each pixel in the image are calculated, and bilinear interpolation is used to obtain the pixel value.

[0137] Step D: Perioral smoothing attenuation weights

[0138] To avoid "vortex" artifacts caused by single-point stretching, a smoothing attenuation weight is applied to the displacement field. The mouth width is defined. This is the horizontal distance between the left and right corner points (points 49 and 55). For any pixel within the perioral region... Its displacement is multiplied by the weight:

[0139]

[0140] This weighting ensures that pixels farther from the healthy side corner of the mouth experience smaller displacements, thus simulating the elastic rheological properties of soft tissue.

[0141] Step E: Quality Control and Dataset Construction

[0142] The generated distorted images were manually and automatically filtered to remove the following samples:

[0143] Cartoon-style or non-human-faced images;

[0144] Samples where text or watermarks cover more than 30% of the area;

[0145] Samples exhibiting global texture corruption or unrecognizable facial structures;

[0146] Samples with inconsistent emotion category labels.

[0147] Approximately 4,500 high-quality distortion samples were obtained, each with a three-level annotation: emotion category (7 categories), HB severity (3 levels), and affected side direction (left / right). To enhance robustness, random illumination perturbations, random occlusion (cutout), and Gaussian noise were superimposed online during training.

[0148] Step F: Construction of the External Validation Set (FPED)

[0149] To verify cross-domain generalization ability, we collected the Facial Droop and FacialParalysis datasets from Kaggle and the FNP Detection dataset from Roboflow, and supplemented them with real facial images of stroke patients from YouTube and medical science popularization platforms. After duplicate sample removal and secondary emotion annotation (independently annotated by two professional annotators, with a third physician making the final decision in case of discrepancies), we obtained approximately 600 real facial paralysis images, which served as an independent external validation set, FPED. This dataset was not used for any training or hyperparameter tuning.

[0150] Example 5: Training Strategies and Knowledge Distillation

[0151] This embodiment describes in detail the training process of the student network (ResNet34_CCA_SK-MSPA), including the loss function, data sampling, optimizer, and regularization techniques.

[0152] (1) Loss function

[0153] The total loss is the weighted sum of three terms:

[0154]

[0155] in: The cross-entropy loss between student network predictions and real hard labels is calculated using class weights. Weighting:

[0156]

[0157] Category weight This is to alleviate the long-tail distribution problem.

[0158] : Distillation loss of soft label, using KL divergence, temperature :

[0159]

[0160] in The logits for the student and teacher networks, respectively. This refers to the Softmax function.

[0161] Spatial attention distillation loss. Extracting high-order feature maps of the teacher and student networks before the classification head. First, calculate the spatial attention map (channel energy at each spatial location):

[0162]

[0163] Then calculate the mean squared error:

[0164]

[0165] This embodiment takes .

[0166] (2) Staged dynamic resampling

[0167] For the first 8 epochs of training, inverse frequency-weighted random sampling was used. Each class... The sampling probability is:

[0168]

[0169] This means that the smaller the sample size of a class, the higher its probability of being sampled. Within each batch, classes are drawn according to this probability distribution, and then samples are randomly selected from those classes. This strategy forces the model to learn the features of minority classes (such as fear and aversion) fully in the early stages.

[0170] After the 9th epoch of training, switch to standard random sampling (based on the original data distribution) to make the model decision boundary converge toward the true posterior probability.

[0171] (3) Optimizer and learning rate scheduling

[0172] AdamW optimizer was used, with the following parameter settings: Weight decay The learning rate rose to [a certain level] after warm-up. .

[0173] Learning rate scheduling strategy: The first two epochs are used for linear warm-up, increasing linearly from 0 to 1. Then cosine annealing is used, eventually decaying to... :

[0174]

[0175] (4) Regularization and acceleration techniques

[0176] Exponential Moving Average (EMA): Maintains an exponential moving average for the model parameters, with a decay rate. Use the EMA parameter during verification and testing.

[0177] Mixed Precision Training (AMP): Using PyTorch's automatic mixed precision, forward propagation and gradient calculation use float16, while sovereign re-update maintains float32, reducing memory usage by about 40% and increasing training speed by about 50%.

[0178] Label smoothing: in calculation When using label smoothing, the smoothing coefficient is... hard label Replaced with To prevent overfitting.

[0179] (5) Summary of training and inference parameters

[0180] Batch size: 32

[0181] Total epochs: 120

[0182] Input image size: 224×224×1 (grayscale) or 224×224×3 (if RGB is used, the original FER2013 grayscale image will be copied to three channels).

[0183] Teacher network: ConvNeXt-Tiny, pre-trained on ImageNet and fine-tuned for 20 epochs on a distorted training set (learning rate 1e-5).

[0184] Hardware: NVIDIA RTX 1650 4GB, training time approximately 6 hours.

[0185] CPU inference: Intel Core i7-10750H, single-frame latency 13.2ms (using ONNX Runtime).

[0186] Example 6: Experimental Results and Comparison

[0187] This embodiment demonstrates the performance of the method of the present invention (Ours) on the self-built distortion FER2013 test set and the real facial paralysis FPED validation set, and compares it with a variety of baseline models.

[0188] (1) Evaluation indicators

[0189] Accuracy: The proportion of correctly classified samples out of the total number of samples.

[0190] Macro-F1 score: The arithmetic mean of the F1 scores for each category is calculated, and it is not affected by class imbalance.

[0191] Parameters: The number of trainable parameters in a model, expressed in millions (M).

[0192] Inference Time: The average time (ms) it takes to process a single frame of an image on the CPU.

[0193] (2) Results of the FER2013 test set with distortion

[0194] Table 2. Comparison of test results for multiple models

[0195] MobileNetV3-Small 62.34 58.12 2.54 9.8 VGGNet 63.87 60.45 138.4 32.5 ResNet50 67.21 63.98 25.6 18.7 POSTER V2 (Transformer) 68.03 65.42 37.8 24.1 Ours (ResNet34+CCA+SK-MSPA+KD) 70.15 67.83 21.69 13.2

[0196] This method outperforms the strongest baseline POSTER V2 by 2.12 percentage points and Macro-F1 by 2.41 percentage points in accuracy; it reduces the number of parameters by 15% compared to ResNet50 and improves inference speed by 29%. Compared to the lightweight MobileNetV3, it improves accuracy by 7.81 percentage points with only an increase of about 19M parameters, but achieves an acceptable performance balance on edge devices.

[0197] (3) Validation results of real facial paralysis FPED (zero-sample transfer)

[0198] Table 3. Comparison of FPED validation set test results for various models

[0199] MobileNetV3-Small 58.23 54.67 ResNet50 61.45 57.89 POSTER V2 62.87 60.12 Ours 65.32 62.58

[0200] Even without having seen real patient images, this method still maintained an accuracy of 65.32%, an improvement of 2.45 percentage points compared to POSTERV2, validating the clinical knowledge-driven simulation strategy and the cross-domain generalization ability of the CCA module.

[0201] (4) Ablation test

[0202] On the FER2013 test set, each core module was gradually removed, and the results are shown in Table 4.

[0203] Table 4. Comparison of FER2013 Distortion Test Results

[0204] Exp1 √ × × × 66.45 64.20 21.28 Exp2 √ √ × × 64.95 63.23 21.44 Exp3 √ × √ × 68.77 66.15 21.53 Exp4 √ √ √ × 69.58 67.29 21.69 Ours √ √ √ √ 70.15 67.83 21.69

[0205] Analysis: Adding CCA alone (Exp2) resulted in a 1.5% decrease in accuracy, indicating that the CCA module alters the feature distribution and needs to be combined with a multi-scale module to be effective. Adding SK-MSPA alone (Exp3) improved accuracy by 2.32%, validating the importance of dynamic receptive fields. The synergy between CCA and SK-MSPA (Exp4) further improved accuracy to 69.58%. Knowledge distillation (Exp5) improved accuracy by another 0.57% with no change in parameters, demonstrating the effectiveness of spatial attention distillation.

[0206] (5) Visualization analysis

[0207] Confusion matrix ( Figure 5 The baseline model (Exp1) suffered from severe confusion in the "fear" and "disgust" categories, with an accuracy of less than 40%. The method of this invention improved the recall rates of these two categories to 58% and 62%, respectively. The recognition rate of the "happiness" category was improved from 89% to 93%, and that of the "sadness" category from 72% to 78%.

[0208] t-SNE feature clustering ( Figure 6 In the baseline model, the feature points of each emotion category are mixed, especially the overlap between "fear" and "sadness". In the method of this invention, each category forms a more compact cluster and the inter-class spacing is significantly increased, indicating that the features extracted by the model have stronger discriminative power.

[0209] Grad-CAM heatmap: The baseline model's heatmap shows scattered activation areas, including irrelevant areas such as the forehead and background; the heatmap of the method of this invention is strictly concentrated on the corners of the mouth and cheeks of the lower half of the face, and the activation intensity differs significantly between the affected and healthy sides (stronger on the affected side), indicating that CCA successfully guides the model to focus on pathology-related areas.

[0210] Example 7: Deployment and Real-Time Testing

[0211] The final model of this method (ONNX format, FP32) was tested on the following edge devices:

[0212] Jetson Nano (2GB): Average inference time 48ms / frame, enabling real-time processing at 20FPS.

[0213] Raspberry Pi 4B (4GB): Average inference time 112ms / frame, approximately 9 FPS.

[0214] Smartphone (Snapdragon 865): Inference via NCNN, average 35ms / frame, approximately 28FPS.

[0215] To meet the needs of real-time emotion monitoring in rehabilitation scenarios.

[0216] Industrial applicability

[0217] The emotional feature extraction method proposed in this invention can be integrated into terminal devices such as rehabilitation robots, smart rehabilitation mirrors, and mobile medical apps to monitor the emotional state of stroke patients in real time and assist in clinical rehabilitation decisions. Furthermore, the CCA module and simulation data generation method can be extended to other medical image analysis tasks with structural asymmetry (such as scoliosis, facial nerve palsy, Bell's palsy, etc.), showing broad industrial application prospects.

[0218] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A healthy-affected side contrast channel attention face emotion feature extraction method, characterized in that, Includes the following steps: Step S1: Obtain the face image to be identified, and input the face image into a pre-trained deep convolutional neural network, wherein the deep convolutional neural network includes a healthy-affected side contrast channel attention module; Step S2: Perform the following operations in the healthy-affected side contrast channel attention module: S2.1 Spatial Splitting and Mirror Alignment: The input feature map is split into left and right half-face features along the mid-sagittal axis of the face, and one side is horizontally flipped to make the anatomical positions of the two sides correspond one-to-one. S2.2 Difference Matrix Encoding: Calculate the feature difference matrix between the left half of the face and the flipped right half of the face. This difference matrix is ​​used to quantify the degree of spatial asymmetry between the left and right sides of the face. The calculation formula is as follows: ,in Features of the left half of the face Features of the right half of the face after flipping. ; S2.3 Horizontal Attention Generation: The difference matrix is ​​pooled in one dimension along the height direction, and horizontal attention weights are generated through nonlinear mapping to suppress asymmetric pathological noise; S2.4 Healthy Side Dynamic Evaluation: The left half-face features, the mirror-aligned right half-face features, and the difference matrix are all subjected to global average pooling. The pooling results are then concatenated and sequentially input into the first fully connected layer, the ReLU activation layer, the second fully connected layer, and the Softmax layer to obtain the probability that the left side is the relatively healthy side and the probability that the right side is the relatively healthy side. Based on these two probabilities, the left half-face features and the mirror-aligned right half-face features are continuously weighted through a healthy-affected side soft routing mechanism to obtain the relatively healthy side features and the relatively affected side features. S2.5 Feature Compensation: Based on the difference between the relative healthy side feature and the relative affected side feature, a healthy side compensation mask is generated. This mask is then used to perform soft-gated fusion of the relative healthy side feature and the relative affected side feature to generate the compensated affected side feature. The calculation formula is as follows: in, Characteristic of the affected side, Features of the healthy side For the convolutional mapping used to generate the healthy side compensation mask, It is the Sigmoid activation function. For health compensation masks with values ​​in the range (0-1), Characteristics of the affected side after compensation; Step S3: Input the feature map processed in step S2 into the subsequent convolutional layer and classifier, and output the emotion classification result of the face image.

2. The method according to claim 1, characterized in that, The deep convolutional neural network also includes an adaptive multi-scale feature fusion module SK-MSPA, which performs the following operations: The input features are reduced in dimensionality by 1×1 convolutions and then fed in parallel into three depthwise separable convolutional branches with different dilation rates, namely: The corresponding receptive fields are 3×3, 5×5, and 7×7, and the output of each branch is denoted as... ; Will Element-by-element addition yields the fusion feature The channel descriptor is obtained by global average pooling. Then, a selection weight vector is generated through the bottleneck fully connected layer. : in Compression ratio ; right Applying Softmax normalization yields the branch weights. And calculate the dynamic fusion output: Will After recovering the channels through pointwise convolution, they are connected to the original input residuals.

3. The method according to claim 1, characterized in that, The deep convolutional neural network adopts a lightweight network based on the ResNet34 skeleton; a healthy-affected side contrast channel attention module is set at the output of Stage2, and the adaptive multi-scale feature fusion module SK-MSPA is set at the output of Stage3.

4. The method according to claim 1, characterized in that, The deep convolutional neural network is trained using a teacher-student knowledge distillation strategy, where the teacher network is ConvNeXt-Tiny and the student network is a lightweight ResNet34 embedded with CCA and SK-MSPA; the distillation loss includes: Soft-label KL divergence loss: ,temperature ; Spatial attention MSE loss: Total loss: .

5. The method according to claim 1, characterized in that, It also includes clinical knowledge-driven data augmentation steps: referring to the House-Brackmann facial paralysis grading standard, the original healthy face image was upsampled using bicubic interpolation, and 68 facial key points were extracted using Dlib, with the intercanthal distance between the two eyes as the basis. As a rigid normalized baseline, the target displacement of key points on the affected side is determined based on the mapping of light, medium, and severe distortion parameters. Thin-plate spline interpolation is then used to solve the global dense deformation field. in Radial basis functions, A smooth, decaying weighting function is applied to the perioral region to eliminate artifacts. 。 6. The method according to claim 1, characterized in that, During the training phase, a phased dynamic resampling strategy is adopted: in the early stage of training, inverse frequency-weighted random sampling is used for each class. The sampling probability is During the later stages of training, standard random sampling was switched to the standard random sampling method, and class-weighted cross-entropy loss was adopted, with class weights... ,in The total number of samples in the training set. For the number of emotion categories, For the first Number of samples in the class.

7. The method according to claim 1, characterized in that, The emotion classification results include seven basic emotions: anger, disgust, fear, happiness, sadness, surprise, and neutrality.

8. The method according to claim 1, characterized in that, During training, an exponential moving average with a decay rate of 0.999 was used; mixed precision training was employed; and label smoothing with a smoothing coefficient of 0.1 was used.

9. The method according to claim 1, characterized in that, The determination rule for the healthy lateral dynamic evaluation in step S2.4 is: to assess the features of the left half of the face. Features of the right half of the face after mirror alignment and regional characteristics of differences After global average pooling, features are concatenated and then passed through a fully connected layer, activation function, and softmax layer to obtain the left-side probability of the healthy side. The probability of the right side being the healthy side The healthy-side soft routing is based on... and The features of the left and right halves of the face are continuously weighted; when it is necessary to output the discrete healthy side orientation, the side with the higher probability is taken as the relative healthy side.

10. The method according to claim 1, characterized in that, The high-order feature maps extracted by the deep convolutional neural network before the classification head are used for spatial attention distillation, which forces the student network to maintain visual focus consistency with the teacher network in the spatial receptive field.

Images