Gradient covariance analysis-based method for identifying abnormal expressions of old people

Abstract

The invention discloses an old people abnormal expression recognition method based on gradient covariance analysis. The method comprises the steps of old people expression data set construction, time sequence optical flow feature-based facial expression region screening, abnormal expression enhancement loss estimation, gradient covariance-based facial expression region feature enhancement, micro-expression model training and micro-expression model testing. Aiming at the problems that the expression movement of the elderly is not obvious and the abnormal expression is easy to neglect, the facial expression key region is positioned by using the time sequence optical flow feature, the abnormal expression is introduced to enhance the loss so as to punish the leak detection condition, the back propagation gradient of the loss and the covariance thereof are calculated, and the region with larger covariance has more obvious feature change, so that the detection accuracy is improved. According to the method, facial expression region features based on gradient covariance are fully considered, key region analysis is enhanced, and the accuracy of abnormal expression recognition of the old people is effectively improved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of artificial intelligence, computer vision, and elderly health monitoring, and in particular to a method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis. This invention is applicable to active elderly health monitoring systems and can accurately identify normal facial expressions, micro-expressions, and abnormal facial expressions in the elderly, providing technical support for early warning of elderly physiological health and emotional care. Background Technology

[0002] With the accelerating aging of the global population and the continuous expansion of my country's elderly population, health monitoring and emotional care for the elderly have become important social issues. Facial expressions, as non-invasive "direct signals" of human emotional state and physiological health, hold special significance for the elderly. Normal facial expressions reflect daily emotional states, micro-expressions can reveal potential psychological fluctuations, while abnormal expressions (such as painful frowning or difficulty breathing with an open mouth) may be early signs of health problems such as cardiovascular and cerebrovascular diseases and neurodegenerative diseases. Therefore, achieving accurate recognition of elderly facial expressions, especially the effective capture of micro-expressions and abnormal expressions, is of significant practical value for building proactive elderly health monitoring systems and improving the quality of elderly care services.

[0003] Chinese Patent Publication No. CN 116935474 A, entitled "A Micro-expression Recognition Method, Device, Electronic Equipment, and Storage Medium," focuses on the shortcoming of temporal resolution in micro-expression recognition. It employs a dynamic visual sensor to acquire high temporal resolution (1μs level) event stream data, simultaneously acquiring video frame sequences. By improving the Transformer model, it extracts image and facial features from the video frames, converts the event stream data into point clouds, and performs spatial structure-preserving processing. Then, it uses weighted scaling and stitching to achieve multimodal fusion, effectively compensating for the insufficient capture of subtle changes in micro-expressions by video frames. However, this patent is not optimized for specific groups (such as the elderly with complex facial features or special groups with strong facial expression deception), lacks targeted noise processing for multimodal data under complex lighting conditions (such as strong light and weak light), and does not link micro-expressions to potential physiological abnormalities. Its applicable scenarios are limited to general micro-expression recognition and are difficult to directly adapt to specific needs such as health monitoring.

[0004] Chinese Patent Publication No. CN 113822192 A, entitled "A Method, Device, and Medium for Emotion Recognition of Detainees Based on Multimodal Feature Fusion of Transformer," addresses the characteristic of detainees faking emotions. It integrates four modalities of data: text (video meeting dialogue), speech (Mel frequency cepstral coefficients), micro-expressions (facial action units), and body movements (18 key points and motion parameters of the upper body skeleton). After unifying the feature dimensions and embedding positional information through a one-dimensional temporal convolutional layer, it utilizes the cross-attention module of a cross-modal Transformer to achieve intermodal information interaction, followed by classification via a self-attention sequence model and a fully connected layer. However, its technical limitations include: body movements only focus on the upper body, failing to cover key emotional expression areas such as the hands and torso; multimodal fusion relies on a fixed cross-attention mapping relationship, without dynamically adjusting modal weights based on individual detainees' expression habits (e.g., some individuals have extremely small body movements); and the training dataset does not sufficiently cover faked samples of complex negative emotions such as resistance and anxiety, leading to a potential decrease in recognition accuracy in extreme scenarios.

[0005] Chinese Patent Publication No. CN 112101096 A, entitled "A Multimodal Fusion Method for Suicidal Sentiment Perception Based on Speech and Micro-expression," aims at non-invasive monitoring of suicidal sentiment. It uses a Kinect with an infrared camera to collect video and audio. Speech features are extracted from content, tone, and speed and transformed using a BP neural network. Micro-expressions are extracted using Gabor wavelets and then dimensionality reduced using linear discriminant analysis. Feature fusion is achieved through LSTM, self-organizing map (SOM), and a compensation layer. Finally, SoftMax classification is used to determine suicidal sentiment. While this patent achieves non-contact data acquisition, it is modally singular (only speech and micro-expressions), failing to incorporate physiological signals more readily reflecting suicidal tendencies, such as heart rate and skin conductance. Although information loss during SOM algorithm normalization is mitigated by a compensation layer, the compensation value is fixed within the range of [-1,1], unable to adapt to the varying intensity of suicidal sentiment expression among individuals. The suicidal sentiment samples do not distinguish between different stages such as "depression" and "collapse," making early warning difficult; it can only identify relatively obvious suicidal sentiment states.

[0006] Therefore, it is urgent to build a multimodal facial expression recognition technology system for the elderly. By optimizing dataset construction, data preprocessing, feature enhancement, and model training strategies, we can make up for the shortcomings of existing technologies, improve the accuracy and practicality of facial expression recognition for the elderly, and provide technical support for elderly health monitoring and emotional care. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis.

[0008] This invention is achieved through the following technical solution: A method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis includes the following steps: S1. Collect facial expression data of the elderly, preprocess the facial expression data, define facial expression category labels, and construct a multimodal elderly facial expression dataset; S2. By calculating the inter-frame differences of video frame sequences in the multimodal elderly facial expression dataset to locate the peak frames of facial expressions, the Gunnar Farneback dense optical flow algorithm is called to extract optical flow features and filter out key region blocks with high correlation to facial expression movements. S3. Design a variant of Focal Loss with class weights as the main loss function, and construct a gradient difference loss function based on the difference in gradient norms between normal and abnormal expression samples. Dynamically adjust the weights of the two according to the training cycle to optimize the model's learning of abnormal expressions. S4. Divide the facial images in the multimodal elderly expression dataset into image blocks, calculate the gradient vector of the image blocks and construct their spatial and temporal covariance matrices, train the prior estimator of the covariance matrix and generate region attention weights, and then enhance the expression region features that integrate gradient variance features and optical flow features. S5. Using the enhanced facial expression region feature map obtained, perform convolution downsampling, global average pooling, layer normalization, and pooling operations in sequence to obtain feature vectors. Input the obtained feature vectors into an MLP layer containing two fully connected layers, and iteratively train the micro-expression model in combination with the loss function of S3. S6. Extract the features of each modality from the test video, and obtain the final recognition result by dynamically weighting and fusing the single-modal recognition results. Then, evaluate and optimize the micro-expression model.

[0009] The specific details of step S1 are as follows: S1-1: Collect multimodal facial expression data; S1-1-1: Determine the age range and select elderly people aged ≥60 years as the data collection subjects; S1-1-2: Refine the stratified screening by gender and age, and further refine the screening based on the distribution of wrinkles and the degree of muscle relaxation. S1-1-2-1: Wrinkle Distribution Screening: Focusing on the core expression areas of the forehead, around the eyes, and around the mouth, a combination of visual scoring and image quantification is used for screening. The classification includes wrinkle location, density, and depth. The density includes: ≤6 wrinkles / cm² for low to medium density, and ≥7 wrinkles / cm² for high density. The depth includes: ≤0.3mm for shallow to moderate, and ≥0.4mm for severe. S1-1-2-2: Muscle relaxation screening: For facial muscles related to facial expression, such as the cheek muscles, orbicularis oris, and orbicularis oculi, a combination of dynamic assessment and instrument detection is used for screening, and the relaxation is classified into three levels: mild, moderate, and severe. S1-1-3: Incorporating diverse samples; S1-1-3-1: Inclusion of healthy elderly: Select elderly people with normal physiological wrinkles and normal muscle movement ability in S1-1-2. The specific criteria are: wrinkles are concentrated in non-expression core areas or low to medium density or shallow to moderate degree, muscles are slightly relaxed, standardized facial expressions are met, smile amplitude is ≥3mm, and eye closure recovery is ≤0.5s, and all gender-age layer combinations are covered. S1-1-3-2: Select elderly people with abnormal wrinkles affecting movement and motor function degeneration in S1-1-2. The specific criteria are: wrinkles covering the core area of ​​facial expression + high density + severe, moderate to severe muscle relaxation, failure to meet standardized facial expression movements, smile amplitude < 1mm, and eye closure recovery > 1s. It also covers all gender-age layer combinations. S1-2: Facial dynamic sequence acquisition; S1-2-1: Select the acquisition device, using a high-definition industrial camera with a resolution ≥1920×1080; S1-2-2: Set the acquisition parameters and fix the camera frame rate at 30fps; S1-2-3: Control the data acquisition environment and conduct data acquisition in a controlled lighting environment with an illuminance of 500-800 lux and no direct strong light; S1-2-4: Clearly define the content to be collected, and collect dynamic sequences of the frontal and 45° side profile of the face, with each sample lasting ≥5 minutes; S1-3: Physiological and speech data acquisition; S1-3-1: Collect electromyographic signals synchronously using a wearable sensor, with a sampling rate of 1000Hz to ensure the sensor fits the facial muscle area. S1-3-2: Collects heart rate and skin conductance signals, and simultaneously collects heart rate variability and skin conductance response. The sensor is attached to the inside of the wrist. S1-3-3: Record audio segments using an omnidirectional microphone, with a sampling rate of 44.1kHz and a quantization bit depth of 16bit; S1-3-4: Collect daily activity data using environmental sensors and wearable devices; S1-4: Cover expression categories, specifically covering normal expressions, micro-expressions, and abnormal expressions; S1-5: Facial image preprocessing; S1-5-1: Denoising and enhancement processing. The nonlocal mean denoising algorithm is used to eliminate Gaussian noise from the facial dynamic sequence acquired in S1-2. Histogram equalization is used to improve the facial texture contrast of the resulting facial dynamic sequence. S1-5-2: Key point localization and normalization: Based on the MTCNN model, 68 facial landmarks are located for the dynamic facial sequence. The image is normalized to 256×256 pixels through affine transformation to ensure that the line connecting the eyes is horizontally centered. S1-5-3: Crop sub-regions. For each frame, crop four sub-regions: left eye, right eye, left lip, and right lip, with a size of 224×224 pixels. S1-6: Electromyography signal preprocessing: first, the signal is filtered by a 50Hz notch filter to remove power frequency interference, then the effective frequency band of 0-200Hz is extracted by the db4 wavelet, and finally the baseline is smoothed by a 1s sliding window. S1-7: Preprocessing of heart rate and skin conductance signals. The heart rate signal is filtered by a 1-second window mean to eliminate fluctuations, and the skin conductance signal is normalized to the [0,1] interval by Z-score. S1-8: Speech data preprocessing, which sequentially performs pre-emphasis, framing, Hanning window processing, and endpoint detection to remove silent segments before extracting 13-dimensional Vimel frequency cepstral coefficients; S1-9: Expression Category Definition; S1-9-1: Definition of normal facial expression: refers to a stable emotional expression lasting ≥2 seconds. Based on the facial movement characteristics, normal facial expressions are classified into 6 categories: calm, smiling, sad, angry, surprised, and disgusted. S1-9-2: Micro-expression definition refers to brief muscle movements lasting 0.1-0.5 seconds. The start and end frames and the areas involved are marked by optical flow. Micro-expressions are classified into 8 categories: blinking, frowning, eyebrow raising, corner of mouth pulling down, nasal flaring, lip pursing, chin slightly raised, and unilateral facial twitching. S1-9-3: Definition of abnormal facial expressions, referring to atypical facial expressions caused by physiological discomfort. Combined with physiological signal fluctuations, abnormal facial expressions are classified into 5 categories: painful frowning, difficulty breathing with open mouth, dizziness with closed eyes, difficulty swallowing with grimacing, and chest tightness with frowning. S1-10: Sample labeling; S1-10-1: Define the total number of categories C as 19, including 6 categories of normal expressions, 8 categories of micro-expressions, and 5 categories of abnormal expressions; S1-10-2: Define the sample size N, which includes healthy elderly samples and elderly samples with abnormal wrinkles and motor function deterioration that affect movement, and both types of samples are fully covered in the gender-age dimension; S1-10-3: Define the true category label y. For the i-th sample and the j-th expression, i=1,2,...,N, j=1,2,...,19, if the true expression of sample i belongs to category j, then label it y. If the true expression of sample i does not belong to category j, that is, to one of the other 18 categories, then it is labeled. In each row of labels corresponding to sample i, there is only one 1, i.e., single-label labeling. Each sample belongs to only one type of expression, with no overlap between multiple categories. The other 18 labels are all 0, satisfying the condition that... ; S1-10-4: Label emotions with tags, each team member independently labels, and consistency is tested using the Kappa coefficient; S1-10-5: Mark micro-expression localization, combining optical flow trajectory tracking and frame-by-frame verification, mark the micro-expression start frame, end frame, duration and involved area; S1-10-6: Associate multimodal labels, link facial expression labels with corresponding time period physiological signals and speech segments to form a multimodal label set; S1-11: Dataset partitioning and storage; S1-11-1: Divide the dataset into training, validation, and test sets in a 7:2:1 ratio to ensure that the distribution of expression categories is consistent across all sets; S1-11-2: Store the dataset. Image data is saved in PNG format, physiological signals and speech features are saved in CSV format, and annotation information is saved in JSON format.

[0010] The specific details of step S2 are as follows: S2-1: Locating the peak frame; S2-1-1: Calculate inter-frame differences. For the facial dynamic video sequence preprocessed in S1-5, iterate through consecutive frames after the starting frame. The formula is as follows: , For subsequent frames, As the starting frame, For histogram statistics, D represents the inter-frame difference between consecutive frames and the starting frame; S2-1-2: Filter peak frames. Select the frames with the largest inter-frame differences as peak frames to anchor the moment when micro-expression muscle movements are most intense. S2-2: Extract optical flow features; S2-2-1: Call the optical flow algorithm, call the Gunnar Farneback dense optical flow algorithm, input a video segment composed of the start frame and the peak frame, and set the pyramid layer number to 5 and the number of iterations to 10; S2-2-2: Extract optical flow components, extract the horizontal optical flow component (u-axis) and the vertical optical flow component (v-axis); S2-2-3: Calculate optical strain according to the formula. x, y are pixel coordinates, u is the horizontal optical flow component, and v is the vertical optical flow component. For light strain; S2-2-4: Construct an optical flow feature map and fuse the u, v, and ε channels to generate a three-dimensional optical flow feature map; S2-3: Critical area processing; S2-3-1: Detect facial key points and use the MTCNN model to identify 81 facial key points in the peak frame; S2-3-2: Crop key areas. Based on the coordinates of key points, crop the left eye, right eye, left lip, and right lip areas, compressing their size to half of the original frame. S2-3-3: Divide the optical flow map into blocks. Divide the clipped key region optical flow map into discrete sub-blocks using a 16×16 pixel grid. S2-4: Valid feature selection; S2-4-1: Calculate the correlation degree of feature blocks. For each 16×16 feature block, use the Pearson correlation coefficient according to the formula. Calculate, where, This represents the i-th pixel sample of the horizontal optical flow component u within the feature block. This represents the pixel mean value of the horizontal optical flow component u within the feature block; This represents the i-th pixel sample of the vertical optical flow component v within the feature block. This represents the pixel mean value of the vertical optical flow component v within the feature block; S2-4-2: Filter valid feature blocks, and filter feature blocks with a correlation degree ≥ 0.7; S2-4-3: Regularize the feature block size. Through bilinear interpolation to complete or trim, regularize the effective feature blocks to a standard size of 16×16.

[0011] The specific details of step S3 are as follows: S3-1: Designing variants of the loss function; S3-1-1: To address the class imbalance problem caused by the scarcity of abnormal facial expression samples, a variant of Focal Loss with class weights is designed; S3-1-2: For the sample labels y, N, C mentioned in step S1-10, calculate the loss function based on the traditional cross-entropy function, the formula is as follows: Introducing category weights Obtain the weighted cross-entropy loss function ; S3-1-3: Introducing Modulation Factor , Construct a Focal Loss variant with class weights: ; S3-1-4: Statistical analysis of the percentage of normal facial expression samples in the training set Micro-expression sample ratio Percentage of abnormal facial expression samples ,according to , , Focus parameters The validation set was determined through a grid search. Among the candidate values ​​[0.5, 1, 1.5, 2], the value that minimized the validation set loss and achieved the highest F1 score for abnormal facial expression recognition was selected. , , , These represent the weights for normal facial expressions, micro-expressions, and abnormal facial expressions, respectively. S3-2: Calculate the gradient norm difference; S3-2-1: Based on the backpropagation algorithm, use automatic differentiation tools to calculate the gradient of the loss function with respect to the model parameters; S3-2-2: Define the gradient norm calculation method. The L2 norm is used to calculate the gradient vector norm. Let the gradient vector of the model parameters be... The dimension is M, where M is the total number of parameters, and the gradient norm is... Among them, the adoption As the k-th component of the gradient vector, calculate the gradient norm of the normal facial expression samples respectively. Gradient norm of abnormal facial expression samples ; S3-2-3: In each training batch, normal expression samples and abnormal expression samples are input into the model separately. The number of normal expression samples is... The number of abnormal expression samples is After forward propagation to calculate the prediction results and loss, the gradient vector of each sample is obtained through backpropagation. Following the gradient norm calculation method described above, the gradient norms of the two classes of samples are calculated separately and averaged. , It continuously tracks and calculates over 50 training cycles; S3-3: Construct the gradient difference loss function; S3-3-1: If and If the difference is small, it means that the model has not effectively distinguished the two types of facial expression features, and this state of indifference needs to be penalized by the gradient difference loss function; if the difference is large, it means that the model has captured the difference between the two types of facial expression features, and the loss is reduced accordingly. S3-3-2: Construct the mathematical expression and use the mean squared error (MSE) as a measure. and The difference is given by the gradient difference loss function: Where K is the number of training batches. are the average gradient norms of normal and abnormal facial expressions in the i-th batch, respectively; S3-3-3: Introducing Weighting Coefficients adjust and The proportion of the total loss function is: Through validation set experiments, it was determined that the candidate values ​​[0.01, 0.05, 0.1, 0.5] are selected to achieve the optimal overall performance of the model. ; S3-4: Dynamically adjust the loss weight; S3-4-1: Design an adjustment mechanism; in the initial training phase, focus on learning the basic features of normal facial expressions. The value should be relatively small; after the model has stably learned normal facial expression features in the later stages of training, the value should be increased. Enhance the learning of abnormal facial expression features; S3-4-2: Determine the basis and method for adjustment, based on the number of training epochs. Let the total training epochs be T, and the initial weights... Adjustment coefficient When epoch hour, ;when hour, ;when hour, ; S3-4-3: Implement the adjustment process. In the training code, after each training epoch, update according to the above rules based on the current epoch, and apply it to the next epoch. The calculation ensures that the model adaptively adjusts the learning weights at different stages.

[0012] The specific details of step S4 are as follows: S4-1: Calculate the gradient vector of the image patch; S4-1-1: Determine the segmentation strategy. For the facial image preprocessed in S1, use non-overlapping fixed-size segmentation to obtain 16×16 pixel image blocks, 16 horizontally and 16 vertically, for a total of 256 blocks. S4-1-2: Select gradient calculation operator. Use the Sobel operator to extract the horizontal and vertical gradients of the image patch, and the horizontal template. , Each 16×16 block is convolved with two templates to obtain the horizontal gradient components. ; S4-1-3: Calculate the gradient vector for pixels within the block. Horizontal gradient ,in The pixel intensity of a facial image patch, where m and n are the coordinate offsets of neighboring pixels relative to the center pixel. The vertical gradient is calculated similarly, and the gradient vector for each patch is... ; S4-2: Construct the gradient covariance matrix; S4-2-1: Define the spatial neighborhood. The spatial neighborhood of each 16×16 block consists of horizontally and vertically adjacent blocks. S4-2-2: Calculate the gradient covariance of the spatial neighborhood. Let the gradient vector of block A be... Neighborhood Block The gradient vector is Spatial covariance matrix The elements are Where N=2, representing the dimension of the gradient vector. , Block A and The gradient mean; K represents the traversal index of the gradient vector. This represents the component vector of the gradient vector of block A in the Kth dimension. Display Block The gradient vector is the component vector in the Kth dimension; S4-2-3: Calculate the temporal inter-frame gradient covariance. For dynamic facial expression videos, calculate the block at time t. With time block t+1 Temporal covariance: ; Represents the target image patch at time t The complete gradient vector, t+1 time block The component of the gradient vector in the k-th dimension, t+1 time block The complete gradient vector, where T is the vector transpose; S4-3: Training the covariance prior estimator; S4-3-1: Construct the model structure, using a multilayer perceptron (MLP) as the covariance prior estimator. The number of nodes in the input layer is 2(k+1), where k is the number of spatial neighborhood blocks. There are 2 hidden layers with 128 and 64 nodes respectively. The activation function is ReLU. The number of nodes in the output layer is the same as that in the input layer. The output is the estimated gradient covariance matrix. S4-3-2: Prepare training data. Extract the gradient covariance matrix of image patches of normal, minor, and abnormal facial expression samples from the training set in step S1 as input and the corresponding facial expression labels as output. Perform data augmentation by randomly flipping and rotating the image patches. S4-3-3: Perform the training process, using cross-entropy loss to measure the difference between the predicted covariance and the true covariance, using the Adam optimizer for training, setting 50 training epochs, evaluating performance on the validation set every 5 epochs and adjusting the number of hidden layer nodes or the learning rate; S4-4: Generate region attention weights; S4-4-1: Calculate the attention score, assuming the estimated covariance of block A is... The prior covariance of normal facial expressions is The difference was calculated using the Frobenius norm: The larger the value of D, the greater the difference between block A and normal facial expression features, and the more likely it is to be a key region of abnormal facial expression; F represents the Frobenius norm; S4-4-2: Generate weights, and normalize the attention scores of all blocks using the SoftMax function. weight N is the total number of blocks, and the weight is... Let i be the attention score for the i-th image patch. Attention score for the j-th image patch; S4-4-3: Adjust the features by weighting and fusing the weights with the feature vectors of the corresponding blocks. The adjusted features are... It is the feature vector of a single 16×16 image patch; S4-5: Feature fusion to enhance expression; S4-5-1: Fusing gradient variance and optical flow features, the gradient variance features and the optical flow features extracted in step S2 are mapped to the same dimension D=256 through a fully connected layer, and then fused using a concatenation method: , in Gradient variance features Characteristics of optical flow This represents the fused feature vector obtained by concatenating gradient variance features and optical flow features; S4-5-2: Enhance the representation of abnormal regions by dividing the fused features into image patches, and multiplying the fused feature sub-vector of each patch by the corresponding attention weight. Obtain the adjusted fusion features Integrate all block features to form an enhanced facial expression region feature map. Block-level fusion of feature vectors.

[0013] The specific details of step S5 are as follows: S5-1: Convolutional downsampling aggregation; S5-1-1: Select parameters, use 3×3 convolution kernels, 64 convolution kernels, and downsample the 256×256×3 facial expression region feature map enhanced in step S4. The size after downsampling is 128×128×64. S5-1-2: Perform the operation by inputting the enhanced facial expression region feature map into the convolutional downsampling layer. The convolutional output is activated by ReLU, as shown in the formula. , in For convolution kernel parameters, As a bias, fine-grained feature aggregation is completed. ; S5-2: Global average pooling; S5-2-1: Determine the method: For the 128×128×64 feature map after convolution downsampling, calculate the average value of all pixels by channel, output one value for each channel, and convert it into a 1×1×64 feature vector; S5-2-2: Perform the operation, setting the pixel value of a certain channel of the feature map to be... 'o' is the height index, 'p' is the width index, and 'c' is the channel index. The global average pooling value for this channel is... Traverse all 64 channels to obtain a 1×1×64 feature vector; S5-3: Layer normalization and pooling operations; S5-3-1: Execution layer normalization, applying the formula to the 1×1×64 feature vector. q represents the feature vector output after layer normalization, and s represents the input features. The mean, For variance, , For learnable parameters, perform layer normalization; S5-3-2: Perform 3×3 max pooling, expand the normalized 1×1×64 feature vector into a 3×3×64 feature map, take the maximum value within the 3×3 window and output it. After pooling, it is still a 1×1×64 feature vector. S5-4: Iterative training of MLP layers; S5-4-1: The MLP layer contains two fully connected layers. The first layer has an input dimension of 64 and an output dimension of 256, with the activation function being ReLU. The second layer has an input dimension of 256 and an output dimension of 19, with the activation function being SoftMax. S5-4-2: Iterative training, inputting the normalized and pooled feature vectors from the layer into the MLP layer to obtain the predicted probability distribution, combined with the total loss function in S3. Calculate the loss and iteratively train using the Adam optimizer.

[0014] The specific details of step S6 are as follows: S6-1: Extract single-modal features. Following the S1 preprocessing procedure, extract single-modal features from the test set's facial images, speech data, and physiological signals respectively. S6-2: Obtain the single-modal recognition results, input the extracted single-modal features into the corresponding sub-models respectively, and obtain the single-modal expression recognition results; S6-3: Dynamic weighted fusion, dynamically allocates weights for facial, voice and physiological modalities based on the characteristics of the test scenario, and performs weighted fusion on the single-modal recognition results to obtain the final recognition result; S6-4: Calculate the evaluation index, calculate the accuracy by the ratio of the number of correctly identified samples to the total number of samples, combine the precision and recall to calculate the F1 score and average F1 score for each type of expression, and calculate the unbiased accuracy based on the average accuracy of each category. S6-5: Optimize the model, filter out abnormal expression recognition error sample cases, analyze the reasons for the errors, and optimize the model's attention weights, feature fusion, or parameter settings based on the reasons.

[0015] The advantages of this invention are: Addressing the problem of subtle facial expressions and easily overlooked abnormal expressions in the elderly, this invention utilizes temporal optical flow features to locate key facial expression regions, introduces an abnormal expression enhancement loss to penalize missed detections, and calculates the backpropagation gradient and covariance of the loss. Regions with larger covariance exhibit more significant feature changes and require attention weighting. This invention fully considers the facial expression region features based on gradient covariance, strengthens key region analysis, and effectively improves the accuracy of abnormal expression recognition in the elderly. Attached Figure Description

[0016] Figure 1 This is a detailed flowchart of step S1 of the present invention; Figure 2 This is a flowchart of step S2 of the present invention; Figure 3 This is a detailed flowchart of step S3 of the present invention; Figure 4 This is a flowchart of step S4 of the present invention; Figure 5 This is a detailed flowchart of step S5 of the present invention; Figure 6 This is a flowchart of step S6 of the present invention. Detailed Implementation

[0017] The method of this invention first constructs an expression dataset covering elderly people in different physiological states through hierarchical screening and multi-dimensional data collection to ensure data representativeness and completeness. Then, it locates key areas of facial expression movement based on temporal optical flow technology to eliminate redundant information interference. Subsequently, it enhances the feature differences of abnormal expressions through gradient covariance analysis and solves the class imbalance problem by combining dynamic weight loss function. Finally, through multi-stage model training and multimodal fusion testing, it achieves accurate recognition of normal expressions, micro-expressions, and abnormal expressions, ultimately providing reliable technical support for health monitoring of the elderly (such as pain warning and abnormal breathing judgment).

[0018] A method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis includes the following steps: S1: Construction of an elderly facial expression dataset; S2: Facial expression region filtering based on temporal optical flow features; S3: Loss estimation for abnormal facial expression enhancement; S4: Facial expression region feature enhancement based on gradient covariance; S5: Micro-expression model training; S6: Micro-expression model test.

[0019] like Figure 1 As shown, the construction of the elderly facial expression dataset in step S1 specifically includes the following steps: S1-1: Collect multimodal facial expression data; S1-1-1: Determine the age range and select elderly people aged ≥60 years as the data collection subjects; S1-1-2: Refine the stratified screening by gender and age (60-70 years old, 71-80 years old, and over 80 years old), and also conduct refined screening based on the distribution of wrinkles and the degree of muscle relaxation; S1-1-2-1: Wrinkle Distribution Screening: Focusing on core expression areas such as the forehead, around the eyes (upper and lower eyelids), and around the mouth (corners of the mouth and base of the nose), a combination of "visual scoring + image quantification" is used for screening. The classification includes wrinkle location (single area / multiple areas), density (≤6 wrinkles / cm² is low to medium density, ≥7 wrinkles / cm² is high density), and depth (≤0.3mm is shallow to moderate, ≥0.4mm is severe). S1-1-2-2: Muscle Relaxation Screening: For facial expression-related muscles such as the cheek muscles, orbicularis oris, and orbicularis oculi, a combination of "dynamic assessment + instrument detection" is used for screening, and the relaxation is classified into three levels: "mild - moderate - severe". This invention focuses on the core areas of facial expressions (forehead, around the eyes, and around the mouth) and expression-related muscles (cheek muscles, orbicularis oris, etc.) to ensure that the facial features of the selected samples are strongly correlated with expression recognition. This invention avoids sample homogenization (such as collecting only young, healthy elderly people), enabling the model to adapt to the facial features of elderly people in different physiological states, and providing differentiated sample support for the subsequent recognition of abnormal expressions (such as atypical expressions caused by muscle relaxation).

[0020] S1-1-3: Incorporate a diverse sample, including healthy elderly people and elderly people with facial motor function decline in the sample scope; S1-1-3-1: Inclusion of healthy elderly: Select elderly people who meet the criteria of "normal physiological wrinkles + normal muscle movement ability" in S1-1-2. The specific criteria are: wrinkles are concentrated in the non-expression core area / low to medium density / moderate, with mild muscle relaxation, standardized facial expression movements meet the standards (smile amplitude ≥3mm, eye closure recovery ≤0.5s), and must cover all stratification combinations of "gender-age"; S1-1-3-2: Select elderly people in S1-1-2 who have “abnormal wrinkles that affect movement + motor function degeneration”. The specific criteria are: wrinkles covering the core area of ​​facial expression + high density + severe, moderate to severe muscle relaxation, and failure to meet the standard of standardized facial expression movements (smile amplitude < 1mm, eye closure recovery > 1s). It is also necessary to cover all layer combinations of “gender-age”.

[0021] This invention incorporates two sample categories: "healthy elderly" and "elderly with facial motor function decline," and both categories cover all stratified combinations of "gender-age," ensuring a balanced sample distribution. This solves the problem of poor model generalization caused by the "single-sample" nature of traditional datasets.

[0022] S1-2: Facial dynamic sequence acquisition; S1-2-1: Select the acquisition device, using a high-definition industrial camera with a resolution ≥1920×1080; S1-2-2: Set the acquisition parameters and fix the camera frame rate at 30fps; S1-2-3: Control the data acquisition environment and conduct data acquisition in a controlled lighting environment with an illuminance of 500-800 lux and no direct strong light; S1-2-4: Clearly define the content to be collected, and collect dynamic sequences of the frontal and 45° side profile of the face, with each sample lasting ≥5 minutes; S1-3: Physiological and speech data acquisition; S1-3-1: Collect electromyographic (EMG) signals synchronously using a wearable sensor, with a sampling rate of 1000Hz to ensure the sensor fits the facial muscle area. S1-3-2: Collects heart rate and skin conductance signals, and simultaneously collects heart rate variability (HRV, sampling rate 250Hz) and skin conductance response (GSR, sampling rate 500Hz). The sensor is attached to the inside of the wrist. S1-3-3: Record audio segments using an omnidirectional microphone, with a sampling rate of 44.1kHz and a quantization bit depth of 16bit; S1-3-4: Collect daily activity data using environmental sensors and wearable devices; S1-4: Cover facial expression categories, specifically covering normal facial expressions (lasting ≥2s), micro-expressions (0.1-0.5s), and abnormal facial expressions (physiological / pathological atypical expressions); S1-5: Facial image preprocessing; S1-5-1: Denoising and enhancement processing, using nonlocal mean denoising algorithm (denoising window 7×7, similarity threshold 0.1) to eliminate Gaussian noise, and improving facial texture contrast through histogram equalization. S1-5-2: Key point localization and normalization. Based on the MTCNN model, 68 facial landmarks are located and the image is normalized to 256×256 pixels through affine transformation to ensure that the line connecting the eyes is horizontally centered. S1-5-3: Crop sub-regions, cropping four sub-regions: left eye, right eye, left lip, and right lip, with a size of 224×224 pixels; S1-6: Electromyography signal preprocessing: first, the signal is filtered by a 50Hz notch filter to remove power frequency interference, then the effective frequency band of 0-200Hz is extracted by db4 wavelet (5-level decomposition), and finally the baseline is smoothed by a 1s sliding window. S1-7: Preprocessing of heart rate and skin conductance signals. The heart rate signal is filtered by a 1-second window mean to eliminate fluctuations, and the skin conductance signal is normalized to the [0,1] interval by Z-score. S1-8: Speech data preprocessing, which sequentially performs pre-emphasis (coefficient 0.97), framing (frame length 20ms, frame shift 10ms), Hanning window processing, endpoint detection to remove silent segments, and then extracts 13 Vimel frequency cepstral coefficients (MFCC). S1-9: Expression Category Definition; S1-9-1: Normal facial expression is defined as a stable emotional expression that lasts for ≥2 seconds, including 6 categories such as calmness and smiling, and specifies the facial movement characteristics of each type of expression (e.g., the upward movement of the corners of the mouth when smiling is ≥3mm). S1-9-2: Micro-expression definition refers to brief muscle movements lasting 0.1-0.5 seconds, including 8 types such as blinking and frowning, and the start and end frames and involved areas are marked by optical flow method; S1-9-3: Definition of abnormal facial expressions, referring to atypical facial expressions caused by physiological discomfort, including 5 categories such as painful frowning and mouth opening due to difficulty breathing, combined with physiological signal fluctuations for auxiliary definition; S1-10: Sample labeling; S1-10-1: Define the total number of categories C as 19, including 6 categories of normal expressions, 8 categories of micro-expressions, and 5 categories of abnormal expressions; S1-10-2: Define the sample size N, which includes "healthy elderly samples" and "elderly samples with facial motor function degeneration" (refer to S1-1-3), and both types of samples are fully covered in the "gender-age" dimension (gender: male / female; age: 60-70 years, 71-80 years, 80 years and above). S1-10-3: Define the true category label y. For the i-th sample (i=1,2,...,N) and the j-th expression class (j=1,2,...,19), if the true expression of sample i belongs to category j, then label it. If the true expression of sample i does not belong to category j (i.e., belongs to one of the other 18 categories), then label it. In each row of labels corresponding to sample i, there is only one "1" (single label annotation, each sample belongs to only one type of expression, with no overlap of multiple categories), and the remaining 18 are all "0", which satisfies the condition that... ; S1-10-4: Establish and train the annotation team. Form a team of 2-3 members with backgrounds in geriatric psychology, and train them to master the definition of facial expression categories and the Label Studio operating procedures. Annotate emotion tags independently, with team members using a Kappa coefficient (≥0.8) to test consistency. Discrepancies in the annotation samples are submitted to experts for review. Independent annotation by team members means that, without communicating with each other or reviewing others' annotation results, each member independently annotates samples in the elderly facial expression dataset (including facial dynamic sequences, physiological signals, and facial expression categories corresponding to speech segments) based on a unified annotation standard. The core of this approach is to eliminate subjective interference among members and ensure the independence and objectivity of the annotation results. S1-10-5: Mark micro-expression localization, combining optical flow trajectory tracking and frame-by-frame verification, mark the micro-expression start frame, end frame, duration and involved area; S1-10-6: Associate multimodal labels, link facial expression labels with corresponding time period physiological signals and speech segments to form a multimodal label set; S1-11: Dataset partitioning and storage; S1-11-1: Divide the dataset into training, validation, and test sets in a 7:2:1 ratio to ensure that the distribution of expression categories is consistent across all sets; S1-11-2: Store the dataset. Image data is saved in PNG format, physiological signals and speech features are saved in CSV format (including timestamps), and annotation information is saved in JSON format (associated with multimodal data IDs).

[0023] This invention focuses on "sample stratification representativeness + multimodal data correlation + label accuracy" as its core objectives, and completes the dataset construction in stages: First, it stratifies by age (≥60 years old, further subdivided into 60-70 years, 71-80 years, and 80 years and above) and gender, and refines the sample selection by combining wrinkle distribution (location, density, and depth of the core facial expression area) and muscle relaxation degree (dynamic state of expression-related muscles), while also including healthy elderly people and elderly people with facial motor function decline to ensure that the sample covers the differences in facial expression ability among the elderly population; Second, it simultaneously collects facial dynamic sequences (capturing visual features of facial expressions), physiological signals (electromyography reflecting facial muscle activity, heart rate / skin conductance reflecting physiological fluctuations related to emotions), voice data (assisting in judging emotional state), and daily activity data to construct a multi-dimensional data correlation of "visual-physiological-voice"; Then, through preprocessing such as denoising, normalization, and feature extraction, it eliminates data noise and dimensional differences, and unifies the data format; Finally, it defines expression categories according to unified specifications (6 categories of normal expressions, 8 categories of micro-expressions, and 5 categories of abnormal expressions). The dataset is divided into training, validation, and test sets in a 7:2:1 ratio to ensure accurate labeling using independent annotation and consistency checks, providing a high-quality and reliable data foundation for subsequent model training.

[0024] like Figure 2 As shown, the facial expression region selection based on temporal optical flow features in step S2 specifically includes the following steps: S2-1: Locating the peak frame; S2-1-1: Calculate inter-frame differences. For the preprocessed facial dynamic video sequence, traverse the consecutive frames after the starting frame and calculate according to the formula... , For subsequent frames, As the starting frame, For histogram statistics; S2-1-2: Filter peak frames. Select the frames with the largest inter-frame differences as peak frames to anchor the moment when micro-expression muscle movements are most intense. By selecting peak frames through inter-frame differences, the "most significant moments" of facial expression movement can be accurately captured (such as the peak of muscle contraction in micro-expressions or the frame with the greatest facial deformation in abnormal expressions), avoiding the subsequent optical flow calculation from covering meaningless static or transitional frames and significantly reducing redundant computation. At the same time, the anchoring of peak frames solves the problem of "short movement time and weak features" in micro-expressions, enabling subsequent optical flow feature extraction to focus on the most recognizable frames and improve feature accuracy. S2-2: Extract optical flow features; S2-2-1: Call the optical flow algorithm, call the Gunnar Farneback dense optical flow algorithm, input a video segment composed of the start frame and the peak frame, and set the pyramid layer number to 5 and the number of iterations to 10; S2-2-2: Extract optical flow components, extract the horizontal optical flow component (u-axis) and the vertical optical flow component (v-axis); S2-2-3: Calculate optical strain according to the formula. , (x, y are pixel coordinates), calculate optical strain; S2-2-4: Construct an optical flow feature map and fuse the u, v, and ε channels to generate a three-dimensional optical flow feature map (size matches the original frame, such as 256×256×3). S2-3: Critical area processing; S2-3-1: Detect facial key points and call the MTCNN model to identify 81 facial key points (including corners of the eyes, corners of the mouth, brow ridges, etc.) in the peak frame. S2-3-2: Crop key areas. Based on the coordinates of key points, crop the left eye (from the brow bone to the cheekbone), right eye, left lip (from the base of the nose to the chin), and right lip areas, compressing the size to 1 / 2 of the original frame (e.g., 128×128×3). S2-3-3: Divide the optical flow map into blocks. Divide the optical flow map of the cropped key area into discrete sub-blocks using a 16×16 pixel grid (e.g., divide each region into 8×8 blocks). S2-4: Valid feature selection; S2-4-1: Calculate the correlation degree of feature blocks. For each 16×16 feature block, use the Pearson correlation coefficient according to the formula. Calculate, where, This represents the i-th pixel sample of the horizontal optical flow component u within the feature block. This represents the pixel mean value of the horizontal optical flow component u within the feature block; This represents the i-th pixel sample of the vertical optical flow component v within the feature block. This represents the pixel mean value of the vertical optical flow component v within the feature block; S2-4-2: Filter valid feature blocks, and filter feature blocks with a correlation degree ≥ 0.7; S2-4-3: Regularize the feature block size. Through bilinear interpolation to complete or trim, regularize the effective feature blocks to a standard size of 16×16.

[0025] By filtering feature blocks with a correlation coefficient ≥ 0.7 using the Pearson correlation coefficient, noise blocks with chaotic optical flow values ​​and no motion significance (such as false optical flow blocks caused by light reflection) can be automatically eliminated, retaining only valid blocks with consistent motion trends (such as the perilip optical flow block when the corners of the mouth turn up). Then, the size is normalized by bilinear interpolation to ensure that all valid blocks have a uniform format, avoiding subsequent feature fusion deviations caused by size differences and improving the stability of model training.

[0026] This invention achieves precise localization of the core facial expression region through four steps: peak frame localization, optical flow feature extraction, key region processing, and effective feature selection. First, the inter-frame differences in the facial dynamic video sequence are calculated (based on histogram statistics), and the frame with the largest difference value is selected as the "peak frame," anchoring the moment of most intense micro-expression muscle movement. Then, the Gunnar Farneback dense optical flow algorithm is invoked, inputting the starting frame and the peak frame to extract horizontal (u-axis), vertical (v-axis) optical flow components and optical strain (ε), fusing the three channels to construct a three-dimensional optical flow feature map, characterizing the direction and degree of facial movement and deformation. Next, MTCNN is used to identify 81 facial key points in the peak frame, and key regions such as the left eye, right eye, left lip, and right lip are cropped based on the key point coordinates (compressed to half the original frame size), and optical flow patches are divided into 16×16 pixel grids. Finally, the Pearson correlation coefficient (measuring the correlation of pixel optical flow values ​​within a patch) is calculated for each patch, and patches with a correlation coefficient ≥ 0.7 are selected. The effective feature blocks are normalized in size through bilinear interpolation to ensure a uniform feature block format and provide focused input for subsequent feature enhancement.

[0027] like Figure 3 As shown, the abnormal expression enhancement loss estimation in step S3 specifically includes the following steps: S3-1: Designing variants of the loss function; S3-1-1: Determine the design direction and design a Focal Loss variant with class weights to address the class imbalance problem caused by the scarcity of abnormal expression samples; S3-1-2: Let the true class label of the sample be... ( (Indicates that sample i belongs to class j, otherwise it is 0). The model predicts the class probability distribution as follows: The total number of categories is The sample size is First, based on traditional cross-entropy loss... Introducing category weights (and category) The weighted cross-entropy loss is obtained. Then introduce a modulation factor ( ), construct a Focal Loss variant with class weights: ; S3-1-3: Determine parameter values ​​and statistically analyze the proportion of normal facial expression samples in the training set. Micro-expression sample ratio Percentage of abnormal facial expression samples ,according to , , Focus parameters The validation set was determined through a grid search. Among the candidate values ​​[0.5, 1, 1.5, 2], the value that minimized the validation set loss and had the highest F1 score for abnormal facial expression recognition (F1 score is the harmonic mean of precision and recall) was selected. ; S3-2: Calculate the gradient norm difference; S3-2-1: Select a gradient calculation method. Based on the backpropagation algorithm, use an automatic differentiation tool to calculate the gradient of the loss function with respect to the model parameters. Taking a convolutional layer as an example, let the input be X, the convolutional kernel be W, the bias be b, and the output after activation be... Gradient calculation follows the chain rule; S3-2-2: Define the gradient norm calculation method. The L2 norm is used to calculate the gradient vector norm. Let the gradient vector of the model parameters be... (Dimension M, where M is the total number of parameters), gradient norm is Among them, the use of This is the k-th component of the gradient vector. Calculate the gradient norm for each normal facial expression sample. Gradient norm of abnormal facial expression samples The formulas are respectively ; S3-2-3: In each training batch, normal expression samples (number) ) and abnormal facial expression samples (number) The samples are input into the model, and after forward propagation to calculate the prediction results and loss, the gradient vector for each sample is obtained through backpropagation. Following the gradient norm calculation method described above, the gradient norms for both classes are calculated and averaged. , It continuously tracks and calculates over 50 training cycles; S3-3: Construct the gradient difference loss function; S3-3-1: Define the construction approach, if and If the difference is small, it means that the model has not effectively distinguished between the two types of facial expression features, and this "indiscriminate" state needs to be penalized by the gradient difference loss function; if the difference is large, it means that the model has captured the difference between the two types of facial expression features, and the loss is reduced accordingly. S3-3-2: Construct a mathematical expression and use mean squared error (MSE) as a measure. and The difference is given by the gradient difference loss function: Where K is the number of training batches. are the average gradient norms of normal and abnormal facial expressions in the i-th batch, respectively; S3-3-3: Introducing Weighting Coefficients adjust and The proportion of the total loss function is: Through validation set experiments, it was determined that the candidate values ​​[0.01, 0.05, 0.1, 0.5] were selected to achieve the optimal overall performance of the model. ; S3-4: Dynamically adjust the loss weight; S3-4-1: Design an adjustment mechanism; in the initial training phase, focus on learning the basic features of normal facial expressions. The value should be relatively small; after the model has stably learned normal facial expression features in the later stages of training, the value should be increased. Enhance the learning of abnormal facial expression features; S3-4-2: Determine the basis and method for adjustment, based on the number of training epochs. Let the total training epochs be T, and the initial weights... Adjustment coefficient When epoch hour, ;when hour, ;when hour, ; S3-4-3: Implement the adjustment process. In the training code, after each training epoch, update according to the above rules based on the current epoch, and apply it to the next epoch. The calculation ensures that the model adaptively adjusts the learning weights at different stages.

[0028] To address the core issue of "class imbalance caused by scarce abnormal expression samples", a composite loss system of "weighted FocalLoss + gradient difference loss + dynamic weight adjustment" is constructed: First, a variant of FocalLoss with class weights is designed. Based on the traditional cross-entropy loss, class weights inversely proportional to the number of class samples are introduced (the proportion of abnormal expression samples is low, so the weight is higher). At the same time, a modulation factor is added to reduce the loss contribution of easily classified samples (such as a large number of normal expressions), guiding the model to focus on difficult-to-classify abnormal expressions. Second, the gradient norms of normal and abnormal expression samples are calculated through backpropagation (based on the L2 norm), and the average gradient norms of the two types of samples in each training batch are statistically calculated to construct a gradient difference loss function. If the difference between the two gradient norms is small (indicating that the model has not effectively distinguished features), the loss penalty is increased through the mean squared error (MSE). If the difference is large, the loss is reduced. Finally, the weight of the gradient difference loss is dynamically adjusted according to the training epoch (epoch). In the initial stage of training (epoch ≤ T / 3), the weight is small, and the model is prioritized to learn the basic features of normal expressions. In the middle stage (T / 3 < epoch ≤ 2T / 3), the weight gradually increases, guiding the model to focus on abnormal expression features. In the later stage (epoch > 2T / 3), the weight is the largest, strengthening the learning of abnormal expression features to ensure that the model is adaptively optimized in different training stages.

[0029] As Figure 4 shown, the gradient covariance-based facial expression region feature enhancement described in step S4 specifically includes the following steps: S4-1: Calculate the gradient vector of the image block; S4-1-1: Determine the partitioning strategy. For the 256×256 pixel facial image preprocessed in S1, non-overlapping fixed-size partitioning is adopted to obtain 16×16 pixel image blocks (16 in the horizontal and vertical directions, a total of 256). S4-1-2: Select the gradient calculation operator. The Sobel operator is used to extract the horizontal and vertical gradients of the image block. The horizontal template , , and each 16×16 block is convolved with the two templates respectively to obtain the horizontal gradient component ; S4-1-3: Calculate the gradient vector. For the pixels within the block, the horizontal gradient , where is the pixel intensity of the facial image block, m and n are the coordinate offsets of the neighboring pixels relative to the central pixel. The vertical gradient is calculated similarly, and the gradient vector of each block is ; S4-2: Construct the gradient covariance matrix; S4-2-1: Define the spatial neighborhood. The spatial neighborhood of each 16×16 block consists of horizontally and vertically adjacent blocks. S4-2-2: Calculate the gradient covariance of the spatial neighborhood. Let the gradient vector of block A be... Neighborhood Block The gradient vector is Spatial covariance matrix The elements are Where N=2 (gradient vector dimension) , Block A and The gradient mean; S4-2-3: Calculate the temporal inter-frame gradient covariance. For dynamic facial expression videos, calculate the block at time t. With time block t+1 Temporal covariance: K represents the traversal index of the gradient vector. This represents the component vector of the gradient vector of block A in the Kth dimension. Display Block The gradient vector is the component vector in the Kth dimension; Represents the target image patch at time t The complete gradient vector, t+1 time block The component of the gradient vector in the k-th dimension, t+1 time block The complete gradient vector, where T is the vector transpose; S4-3: Training the covariance prior estimator; S4-3-1: Construct the model structure, using a multilayer perceptron (MLP) as the covariance prior estimator. The number of nodes in the input layer is 2(k+1) (k is the number of spatial neighborhood blocks), and there are 2 hidden layers (128 and 64 nodes, with ReLU activation function). The number of nodes in the output layer is the same as that in the input layer, and the output is the estimated gradient covariance matrix. S4-3-2: Prepare training data. Extract the gradient covariance matrix of image patches from the S1 training set for normal, minor, and abnormal facial expression samples as input and the corresponding facial expression labels as output. Perform data augmentation by randomly flipping and rotating the image patches. S4-3-3: Perform the training process, using cross-entropy loss to measure the difference between the predicted covariance and the true covariance, using the Adam optimizer (learning rate 0.001) for training, setting 50 training epochs, evaluating performance on the validation set every 5 epochs and adjusting the number of hidden layer nodes or the learning rate; S4-4: Generate region attention weights; S4-4-1: Calculate the attention score, assuming the estimated covariance of block A is... The prior covariance of normal facial expressions is The difference was calculated using the Frobenius norm: The larger D is, the greater the difference between block A and normal facial expression features, and the more likely it is to be a key area of ​​abnormal facial expression. F represents the Frobenius norm. S4-4-2: Generate weights, and normalize the attention scores of all blocks using the SoftMax function. weight (N is the total number of blocks), weight Let i be the attention score for the i-th image patch. Attention score for the j-th image patch; S4-4-3: Adjust the features by weighting and fusing the weights with the feature vectors of the corresponding blocks. The adjusted features are... It is the feature vector of a single 16×16 image patch; S4-5: Feature fusion to enhance expression; S4-5-1: Integrate gradient variance and optical flow features, and combine gradient variance features (dimension...) Optical flow features extracted from S2 (dimensions) Mapped to the same dimension D=256 through a fully connected layer, and then merged using a stitching method: , ,in Gradient variance features Characteristics of optical flow This represents the fused feature vector obtained by concatenating gradient variance features and optical flow features; S4-5-2: Enhance the representation of abnormal regions by dividing the fused features into image patches, and multiplying the fused feature sub-vector of each patch by the corresponding attention weight. Obtain the adjusted fusion features Integrate all block features to form an enhanced facial expression region feature map. Block-level fusion of feature vectors.

[0030] This invention focuses on "enhancing the feature differences between abnormal and normal facial expressions." It achieves feature enhancement through five steps: gradient vector calculation, covariance matrix construction, covariance prior estimation, attention weight generation, and feature fusion. First, the preprocessed 256×256 facial image is divided into non-overlapping 16×16 blocks. The Sobel operator is used to extract the horizontal and vertical gradient vectors of each block, characterizing the edge changes of pixels within the block (such as wrinkles and muscle movement edge features). Then, two types of covariance matrices are constructed— Spatial neighborhood gradient covariance (calculating the gradient correlation between adjacent blocks within the same frame, reflecting the spatial distribution characteristics of facial expressions, such as eye-lip coordination) and temporal inter-frame gradient covariance (calculating the gradient change of the same block between different frames, reflecting the temporal dynamic characteristics of facial expressions, such as the gradient evolution during the duration of facial expressions) are used. A multilayer perceptron (MLP) is then used to train a covariance prior estimator, taking the gradient covariance matrices of normal, minor, and abnormal facial expressions in the training set as input and facial expression labels as output. Data augmentation (random flipping and rotating image patches) is used to improve the generalization of the estimator. Next, the Frobenius norm difference between the estimated covariance of the block to be identified and the prior covariance of normal facial expressions is calculated (the larger the difference, the more likely it is a key region of an abnormal facial expression). SoftMax normalization is used to generate region attention weights, which are then used to adjust the feature vectors of the corresponding blocks. Finally, the gradient variance features and the optical flow features extracted by S2 are mapped to the same dimension (D=256) through a fully connected layer, concatenated and fused, and multiplied by the attention weights for each image patch to form an enhanced facial expression region feature map, strengthening the feature representation of abnormal regions.

[0031] like Figure 5 As shown, the micro-expression model training described in step S5 specifically includes the following steps: S5-1: Convolutional downsampling aggregation; S5-1-1: Select parameters, use 3×3 convolution kernels (stride 2), 64 convolution kernels, downsample the 256×256×3 feature map enhanced by S4, and the size after downsampling is 128×128×64; S5-1-2: Perform the operation by inputting the enhanced feature map into the convolutional downsampling layer. The convolutional output is activated by ReLU, as shown in the formula. , in For convolution kernel parameters, As a bias, fine-grained feature aggregation is completed. ; S5-2: Global average pooling; S5-2-1: Determine the method: For the 128×128×64 feature map after convolution downsampling, calculate the average value of all pixels by channel, output one value for each channel, and convert it into a 1×1×64 feature vector; S5-2-2: Perform the operation, setting the pixel value of a certain channel of the feature map to be... 'o' is the height index, 'p' is the width index, and 'c' is the channel index. The global average pooling value for this channel is... Traverse all 64 channels to obtain a 1×1×64 feature vector; S5-3: Layer normalization and pooling operations; S5-3-1: Execution layer normalization, applying the formula to the 1×1×64 feature vector. q represents the feature vector output after layer normalization, and s represents the input features. The mean, For variance, , For learnable parameters, perform layer normalization; S5-3-2: Perform 3×3 max pooling, expand the normalized 1×1×64 feature vector into a 3×3×64 feature map, take the maximum value within the 3×3 window and output it. After pooling, it is still a 1×1×64 feature vector. S5-4: Iterative training of MLP layers; S5-4-1: Design structure, the MLP layer contains two fully connected layers. The first layer has an input dimension of 64 and an output dimension of 256 (activation function ReLU). The second layer has an input dimension of 256 and an output dimension of 19 (consistent with the number of expression categories, activation function SoftMax). S5-4-2: Iterative training, inputting the normalized and pooled feature vectors from the layer into the MLP layer to obtain the predicted probability distribution, combined with the total loss function in S3. Calculate the loss and iteratively train using the Adam optimizer (learning rate 0.001).

[0032] like Figure 6 As shown, the micro-expression model test described in step S6 specifically includes the following steps: S6-1: Extract single-modal features. Following the S1 preprocessing procedure, extract single-modal features from the test set's facial images, speech data, and physiological signals respectively. S6-2: Obtain the single-modal recognition results, input the extracted single-modal features into the corresponding sub-models respectively, and obtain the single-modal expression recognition results; S6-3: Dynamic weighted fusion, dynamically allocates weights for facial, voice and physiological modalities based on the characteristics of the test scenario, and performs weighted fusion on the single-modal recognition results to obtain the final recognition result; S6-4: Calculate the evaluation index, calculate the accuracy by the ratio of the number of correctly identified samples to the total number of samples, combine the precision and recall to calculate the F1 score and average F1 score for each type of expression, and calculate the unbiased accuracy (UAR) based on the average accuracy of each category. S6-5: Optimize the model, filter out abnormal expression recognition error sample cases, analyze the reasons for the errors (such as feature extraction, fusion or parameter problems), and optimize the model attention weights, feature fusion or parameter settings according to the reasons.

Claims

1. A method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis, characterized in that, Includes the following steps: S1. Collect facial expression data of the elderly, preprocess the facial expression data, define facial expression category labels, and construct a multimodal elderly facial expression dataset; S2. By calculating the inter-frame differences of video frame sequences in the multimodal elderly facial expression dataset to locate the peak frames of facial expressions, the Gunnar Farneback dense optical flow algorithm is called to extract optical flow features and filter out key region blocks with high correlation to facial expression movements. S3. Design a variant of Focal Loss with class weights as the main loss function, and construct a gradient difference loss function based on the difference in gradient norms between normal and abnormal expression samples. Dynamically adjust the weights of the two according to the training cycle to optimize the model's learning of abnormal expressions. S4. Divide the facial images in the multimodal elderly expression dataset into image blocks, calculate the gradient vector of the image blocks and construct their spatial and temporal covariance matrices, train the prior estimator of the covariance matrix and generate region attention weights, and then enhance the expression region features that integrate gradient variance features and optical flow features. S5. Using the enhanced facial expression region feature map obtained, perform convolution downsampling, global average pooling, layer normalization, and pooling operations in sequence to obtain feature vectors. Input the obtained feature vectors into an MLP layer containing two fully connected layers, and iteratively train the micro-expression model using the loss function in S3. S6. Extract the features of each modality from the test video, and obtain the final recognition result by dynamically weighting and fusing the single-modal recognition results. Then, evaluate and optimize the micro-expression model.

2. The method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis according to claim 1, characterized in that, The specific details of step S1 are as follows: S1-1: Collect multimodal facial expression data; S1-1-1: Determine the age range and select elderly people aged ≥60 years as the data collection subjects; S1-1-2: Refine the stratified screening by gender and age, and further refine the screening based on the distribution of wrinkles and the degree of muscle relaxation. S1-1-2-1: Wrinkle Distribution Filtering: Focusing on the core expression areas of the forehead, around the eyes, and around the mouth, a combination of visual scoring and image quantification is used for filtering. Classification includes wrinkle location, density, and depth. The density includes: ≤6 wrinkles / cm². 2 Medium to low density, ≥7 strands / cm 2 For high density, the depth includes: ≤0.3mm for shallow to medium density, and ≥0.4mm for heavy density; S1-1-2-2: Muscle relaxation screening: For facial muscles related to facial expression, such as the cheek muscles, orbicularis oris, and orbicularis oculi, a combination of dynamic assessment and instrument detection is used for screening, and the relaxation is classified into three levels: mild, moderate, and severe. S1-1-3: Incorporating diverse samples; S1-1-3-1: Inclusion of healthy elderly: Select elderly people with normal physiological wrinkles and normal muscle movement ability in S1-1-2. The specific criteria are: wrinkles are concentrated in non-expression core areas or low to medium density or shallow to moderate degree, muscles are slightly relaxed, standardized facial expressions are met, smile amplitude is ≥3mm, and eye closure recovery is ≤0.5s, and all gender-age layer combinations are covered. S1-1-3-2: Select elderly people with abnormal wrinkles affecting movement and motor function degeneration in S1-1-2. The specific criteria are: wrinkles covering the core area of ​​facial expression + high density + severe, moderate to severe muscle relaxation, failure to meet standardized facial expression movements, smile amplitude < 1mm, and eye closure recovery > 1s. It also covers all gender-age layer combinations. S1-2: Facial dynamic sequence acquisition; S1-2-1: Select the acquisition device, using a high-definition industrial camera with a resolution ≥1920×1080; S1-2-2: Set the acquisition parameters and fix the camera frame rate at 30fps; S1-2-3: Control the data acquisition environment and conduct data acquisition in a controlled lighting environment with an illuminance of 500-800 lux and no direct strong light; S1-2-4: Clearly define the content to be collected, and collect dynamic sequences of the frontal and 45° side profile of the face, with each sample lasting ≥5 minutes; S1-3: Physiological and speech data acquisition; S1-3-1: Collect electromyographic signals synchronously using a wearable sensor, with a sampling rate of 1000Hz to ensure the sensor fits the facial muscle area. S1-3-2: Collects heart rate and skin conductance signals, and simultaneously collects heart rate variability and skin conductance response. The sensor is attached to the inside of the wrist. S1-3-3: Record audio segments using an omnidirectional microphone, with a sampling rate of 44.1kHz and a quantization bit depth of 16bit; S1-3-4: Collect daily activity data using environmental sensors and wearable devices; S1-4: Cover expression categories, specifically covering normal expressions, micro-expressions, and abnormal expressions; S1-5: Facial image preprocessing; S1-5-1: Denoising and enhancement processing. The nonlocal mean denoising algorithm is used to eliminate Gaussian noise from the facial dynamic sequence acquired in S1-2. Histogram equalization is used to improve the facial texture contrast of the resulting facial dynamic sequence. S1-5-2: Key point localization and normalization: Based on the MTCNN model, 68 facial landmarks are located for the dynamic facial sequence. The image is normalized to 256×256 pixels through affine transformation to ensure that the line connecting the eyes is horizontally centered. S1-5-3: Crop sub-regions. For each frame, crop four sub-regions: left eye, right eye, left lip, and right lip. Set the size to 224×224 pixels. S1-6: Electromyography signal preprocessing: first, the signal is filtered by a 50Hz notch filter to remove power frequency interference, then the effective frequency band of 0-200Hz is extracted by the db4 wavelet, and finally the baseline is smoothed by a 1s sliding window. S1-7: Preprocessing of heart rate and skin conductance signals. The heart rate signal is filtered by a 1-second window mean to eliminate fluctuations, and the skin conductance signal is normalized to the [0,1] interval by Z-score. S1-8: Speech data preprocessing, which sequentially performs pre-emphasis, framing, Hanning window processing, and endpoint detection to remove silent segments before extracting 13-dimensional Vimel frequency cepstral coefficients; S1-9: Expression Category Definition; S1-9-1: Definition of normal facial expression: refers to a stable emotional expression lasting ≥2 seconds. Based on the facial movement characteristics, normal facial expressions are classified into 6 categories: calm, smiling, sad, angry, surprised, and disgusted. S1-9-2: Micro-expression definition refers to brief muscle movements lasting 0.1-0.5 seconds. The start and end frames and the areas involved are marked by optical flow. Micro-expressions are classified into 8 categories: blinking, frowning, eyebrow raising, corner of mouth pulling down, nasal flaring, lip pursing, chin slightly raised, and unilateral facial twitching. S1-9-3: Definition of abnormal facial expressions, referring to atypical facial expressions caused by physiological discomfort. Combined with physiological signal fluctuations, abnormal facial expressions are classified into 5 categories: painful frowning, difficulty breathing with open mouth, dizziness with closed eyes, difficulty swallowing with grimacing, and chest tightness with frowning. S1-10: Sample labeling; S1-10-1: Define the total number of categories C as 19, including 6 categories of normal expressions, 8 categories of micro-expressions, and 5 categories of abnormal expressions; S1-10-2: Define the sample size N, which includes healthy elderly samples and elderly samples with abnormal wrinkles and motor function deterioration that affect movement, and both types of samples are fully covered in the gender-age dimension; S1-10-3: Define the true category label y. For the i-th sample and the j-th expression, i=1,2,...,N, j=1,2,...,19, if the true expression of sample i belongs to category j, then label it y. If the true expression of sample i does not belong to category j, that is, to one of the other 18 categories, then it is labeled. In each row of labels corresponding to sample i, there is only one 1, i.e., single-label labeling. Each sample belongs to only one type of expression, with no overlap between multiple categories. The other 18 labels are all 0, satisfying the condition that... ; S1-10-4: Label emotions with tags, each team member independently labels, and consistency is tested using the Kappa coefficient; S1-10-5: Mark micro-expression localization, combining optical flow trajectory tracking and frame-by-frame verification, mark the micro-expression start frame, end frame, duration and involved area; S1-10-6: Associate multimodal labels, link facial expression labels with corresponding time period physiological signals and speech segments to form a multimodal label set; S1-11: Dataset partitioning and storage; S1-11-1: Divide the dataset into training, validation, and test sets in a 7:2:1 ratio to ensure that the distribution of expression categories is consistent across all sets; S1-11-2: Store the dataset. Image data is saved in PNG format, physiological signals and speech features are saved in CSV format, and annotation information is saved in JSON format.

3. The method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis according to claim 2, characterized in that, The specific details of step S2 are as follows: S2-1: Locating the peak frame; S2-1-1: Calculate inter-frame differences. For the facial dynamic video sequence preprocessed in S1-5, iterate through consecutive frames after the starting frame. The formula is as follows: , For subsequent frames, As the starting frame, For histogram statistics, D represents the inter-frame difference between consecutive frames and the starting frame; S2-1-2: Filter peak frames. Select the frames with the largest inter-frame differences as peak frames to anchor the moment when micro-expression muscle movements are most intense. S2-2: Extract optical flow features; S2-2-1: Call the optical flow algorithm, call the Gunnar Farneback dense optical flow algorithm, input a video segment composed of the start frame and the peak frame, and set the pyramid layer number to 5 and the number of iterations to 10; S2-2-2: Extract optical flow components, extract the horizontal optical flow component (u-axis) and the vertical optical flow component (v-axis); S2-2-3: Calculate optical strain according to the formula. x, y are pixel coordinates, u is the horizontal optical flow component, and v is the vertical optical flow component. For light strain; S2-2-4: Construct an optical flow feature map and fuse the u, v, and ε channels to generate a three-dimensional optical flow feature map; S2-3: Critical area processing; S2-3-1: Detect facial key points and use the MTCNN model to identify 81 facial key points in the peak frame; S2-3-2: Crop key areas. Based on the coordinates of key points, crop the left eye, right eye, left lip, and right lip areas, compressing their size to half of the original frame. S2-3-3: Divide the optical flow map into blocks. Divide the clipped key region optical flow map into discrete sub-blocks using a 16×16 pixel grid. S2-4: Valid feature selection; S2-4-1: Calculate the correlation degree of feature blocks. For each 16×16 feature block, use the Pearson correlation coefficient according to the formula. Calculate, where, This represents the i-th pixel sample of the horizontal optical flow component u within the feature block. This represents the pixel mean value of the horizontal optical flow component u within the feature block; This represents the i-th pixel sample of the vertical optical flow component v within the feature block. This represents the pixel mean value of the vertical optical flow component v within the feature block; S2-4-2: Filter valid feature blocks, and filter feature blocks with a correlation degree ≥ 0.7; S2-4-3: Regularize the feature block size. Through bilinear interpolation to complete or trim, regularize the effective feature blocks to a standard size of 16×16.

4. The method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis according to claim 3, characterized in that, The specific details of step S3 are as follows: S3-1: Designing variants of the loss function; S3-1-1: To address the class imbalance problem caused by the scarcity of abnormal facial expression samples, a FocalLoss variant with class weights is designed; S3-1-2: For the sample labels y, N, C mentioned in step S1-10, calculate the loss function based on the traditional cross-entropy function, the formula is as follows: Introducing category weights Obtain the weighted cross-entropy loss function ; S3-1-3: Introducing Modulation Factor , Construct a Focal Loss variant with class weights: ; S3-1-4: Statistical analysis of the percentage of normal facial expression samples in the training set Micro-expression sample ratio Percentage of abnormal facial expression samples ,according to , , Focus parameters The validation set was determined through a grid search. Among the candidate values ​​[0.5, 1, 1.5, 2], the value that minimized the validation set loss and achieved the highest F1 score for abnormal facial expression recognition was selected. , , , These represent the weights for normal facial expressions, micro-expressions, and abnormal facial expressions, respectively. S3-2: Calculate the gradient norm difference; S3-2-1: Based on the backpropagation algorithm, use automatic differentiation tools to calculate the gradient of the loss function with respect to the model parameters; S3-2-2: Define the gradient norm calculation method. The L2 norm is used to calculate the gradient vector norm. Let the gradient vector of the model parameters be... The dimension is M, where M is the total number of parameters, and the gradient norm is... Among them, the use of As the k-th component of the gradient vector, calculate the gradient norm of the normal facial expression samples respectively. Gradient norm of abnormal facial expression samples ; S3-2-3: In each training batch, normal expression samples and abnormal expression samples are input into the model separately. The number of normal expression samples is... The number of abnormal expression samples is After forward propagation to calculate the prediction results and loss, the gradient vector of each sample is obtained through backpropagation. Following the gradient norm calculation method described above, the gradient norms of the two classes of samples are calculated separately and averaged. , It continuously tracks and calculates over 50 training cycles; S3-3: Construct the gradient difference loss function; S3-3-1: If and If the difference is small, it means that the model has not effectively distinguished the two types of facial expression features, and this state of indifference needs to be penalized by the gradient difference loss function; if the difference is large, it means that the model has captured the difference between the two types of facial expression features, and the loss is reduced accordingly. S3-3-2: Construct the mathematical expression and use the mean squared error (MSE) as a measure. and The difference is given by the gradient difference loss function: Where K is the number of training batches. are the average gradient norms of normal and abnormal facial expressions in the i-th batch, respectively; S3-3-3: Introducing Weighting Coefficients adjust and The proportion of the total loss function is: Through validation set experiments, it was determined that the candidate values ​​[0.01, 0.05, 0.1, 0.5] are selected to achieve the optimal overall performance of the model. ; S3-4: Dynamically adjust the loss weight; S3-4-1: Design an adjustment mechanism; in the initial training phase, focus on learning the basic features of normal facial expressions. The value should be relatively small; after the model has stably learned normal facial expression features in the later stages of training, the value should be increased. Enhance the learning of abnormal facial expression features; S3-4-2: Determine the basis and method for adjustment, based on the number of training epochs. Let the total training epochs be T, and the initial weights... Adjustment coefficient When epoch hour, ;when hour, ;when hour, ; S3-4-3: Implement the adjustment process. In the training code, after each training epoch, update according to the above rules based on the current epoch, and apply it to the next epoch. The calculation ensures that the model adaptively adjusts the learning weights at different stages.

5. The method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis according to claim 4, characterized in that, The specific details of step S4 are as follows: S4-1: Calculate the gradient vector of the image patch; S4-1-1: Determine the segmentation strategy. For the facial image preprocessed in S1, use non-overlapping fixed-size segmentation to obtain 16×16 pixel image blocks, 16 horizontally and 16 vertically, for a total of 256 blocks. S4-1-2: Select gradient calculation operator. Use the Sobel operator to extract the horizontal and vertical gradients of the image patch, and the horizontal template. , Each 16×16 block is convolved with two templates to obtain the horizontal gradient components. ; S4-1-3: Calculate the gradient vector for pixels within the block. Horizontal gradient ,in The pixel intensity of a facial image patch, where m and n are the coordinate offsets of neighboring pixels relative to the center pixel. The vertical gradient is calculated similarly, and the gradient vector for each patch is... ; S4-2: Construct the gradient covariance matrix; S4-2-1: Define the spatial neighborhood. The spatial neighborhood of each 16×16 block consists of horizontally and vertically adjacent blocks. S4-2-2: Calculate the gradient covariance of the spatial neighborhood. Let the gradient vector of block A be... Neighborhood Block The gradient vector is Spatial covariance matrix The elements are Where N=2, representing the dimension of the gradient vector. , Block A and The gradient mean; K represents the traversal index of the gradient vector. This represents the component vector of the gradient vector of block A in the Kth dimension. Display Block The gradient vector is the component vector in the Kth dimension; S4-2-3: Calculate the temporal inter-frame gradient covariance. For dynamic facial expression videos, calculate the block at time t. With time block t+1 Temporal covariance: ; Represents the target image patch at time t The complete gradient vector, t+1 time block The component of the gradient vector in the k-th dimension, t+1 time block The complete gradient vector, where T is the vector transpose; S4-3: Training the covariance prior estimator; S4-3-1: Construct the model structure, using a multilayer perceptron (MLP) as the covariance prior estimator. The number of nodes in the input layer is 2(k+1), where k is the number of spatial neighborhood blocks. There are 2 hidden layers with 128 and 64 nodes respectively. The activation function is ReLU. The number of nodes in the output layer is the same as that in the input layer. The output is the estimated gradient covariance matrix. S4-3-2: Prepare training data. Extract the gradient covariance matrix of image patches of normal, minor, and abnormal facial expression samples from the training set in step S1 as input and the corresponding facial expression labels as output. Perform data augmentation by randomly flipping and rotating the image patches. S4-3-3: Perform the training process, using cross-entropy loss to measure the difference between the predicted covariance and the true covariance, using the Adam optimizer for training, setting 50 training epochs, evaluating performance on the validation set every 5 epochs and adjusting the number of hidden layer nodes or the learning rate; S4-4: Generate region attention weights; S4-4-1: Calculate the attention score, assuming the estimated covariance of block A is... The prior covariance of normal facial expressions is The difference was calculated using the Frobenius norm: The larger D is, the greater the difference between block A and normal facial expression features, and the more likely it is to be a key area of ​​abnormal facial expression. F represents the Frobenius norm. S4-4-2: Generate weights, and normalize the attention scores of all blocks using the SoftMax function. weight N is the total number of blocks, and the weight is... Let i be the attention score for the i-th image patch. Attention score for the j-th image patch; S4-4-3: Adjust the features by weighting and fusing the weights with the feature vectors of the corresponding blocks. The adjusted features are... It is the feature vector of a single 16×16 image patch; S4-5: Feature fusion to enhance expression; S4-5-1: Fusing gradient variance and optical flow features, the gradient variance features and the optical flow features extracted in step S2 are mapped to the same dimension D=256 through a fully connected layer, and then fused using a concatenation method: , in Gradient variance features Characteristics of optical flow This represents the fused feature vector obtained by concatenating gradient variance features and optical flow features through a splicing method. S4-5-2: Enhance the representation of abnormal regions by dividing the fused features into image patches, and multiplying the fused feature sub-vector of each patch by the corresponding attention weight. Obtain the adjusted fusion features Integrate all block features to form an enhanced facial expression region feature map. Block-level fusion of feature vectors.

6. The method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis according to claim 5, characterized in that, The specific details of step S5 are as follows: S5-1: Convolutional downsampling aggregation; S5-1-1: Select parameters, use 3×3 convolution kernels, 64 convolution kernels, and downsample the 256×256×3 facial expression region feature map enhanced in step S4. The size after downsampling is 128×128×64. S5-1-2: Perform the operation by inputting the enhanced facial expression region feature map into the convolutional downsampling layer. The convolutional output is activated by ReLU, as shown in the formula. , in For convolution kernel parameters, As a bias, fine-grained feature aggregation is completed. ; S5-2: Global average pooling; S5-2-1: Determine the method: For the 128×128×64 feature map after convolution downsampling, calculate the average value of all pixels by channel, output one value for each channel, and convert it into a 1×1×64 feature vector; S5-2-2: Perform the operation, setting the pixel value of a certain channel of the feature map to be... 'o' is the height index, 'p' is the width index, and 'c' is the channel index. The global average pooling value for this channel is... Traverse all 64 channels to obtain a 1×1×64 feature vector; S5-3: Layer normalization and pooling operations; S5-3-1: Execution layer normalization, applying the formula to the 1×1×64 feature vector. q represents the feature vector output after layer normalization, and s represents the input features. The mean, For variance, , For learnable parameters, perform layer normalization; S5-3-2: Perform 3×3 max pooling, expand the normalized 1×1×64 feature vector into a 3×3×64 feature map, take the maximum value within the 3×3 window and output it. After pooling, it is still a 1×1×64 feature vector. S5-4: Iterative training of MLP layers; S5-4-1: The MLP layer contains two fully connected layers. The first layer has an input dimension of 64 and an output dimension of 256, with the activation function being ReLU. The second layer has an input dimension of 256 and an output dimension of 19, with the activation function being SoftMax. S5-4-2: Iterative training, inputting the normalized and pooled feature vectors from the layer into the MLP layer to obtain the predicted probability distribution, combined with the total loss function in S3. Calculate the loss and iteratively train using the Adam optimizer.

7. The method for recognizing abnormal facial expressions in the elderly based on gradient covariance analysis according to claim 6, characterized in that, The specific details of step S6 are as follows: S6-1: Extract single-modal features. Following the S1 preprocessing procedure, extract single-modal features from the test set's facial images, speech data, and physiological signals respectively. S6-2: Obtain the single-modal recognition results, input the extracted single-modal features into the corresponding sub-models respectively, and obtain the single-modal expression recognition results; S6-3: Dynamic weighted fusion, dynamically allocates weights for facial, voice and physiological modalities based on the characteristics of the test scenario, and performs weighted fusion on the single-modal recognition results to obtain the final recognition result; S6-4: Calculate the evaluation index, calculate the accuracy by the ratio of the number of correctly identified samples to the total number of samples, combine the precision and recall to calculate the F1 score and average F1 score for each type of expression, and calculate the unbiased accuracy based on the average accuracy of each category. S6-5: Optimize the model, filter out abnormal expression recognition error sample cases, analyze the reasons for the errors, and optimize the model's attention weights, feature fusion, or parameter settings based on the reasons.

Images