Facial depression auxiliary detection method based on dynamic deformable convolution
By constructing a dual-stream DCNN network model integrating HD3 dynamic convolution modules, the problems of weak feature extraction ability and insufficient robustness of existing facial recognition models for depression under facial occlusion are solved, achieving higher inference accuracy and stronger generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHONGSHAN INST OF CHANGCHUN UNIV OF SCI & TECH
- Filing Date
- 2026-04-16
- Publication Date
- 2026-07-21
AI Technical Summary
Existing deep learning-based facial recognition models for depression suffer from weak feature extraction, low inference accuracy, weak generalization ability, and insufficient robustness when faced with facial occlusion caused by head posture, accessories, gestures, or environmental factors.
A facial depression-assisted detection method based on dynamic deformable convolution is adopted. By constructing a dual-stream DCNN network model integrating HD3 dynamic convolution module, combining spatial appearance flow and temporal dynamic flow, feature extraction is performed using RGB image frames and optical flow feature maps, and five-fold cross-validation and two-stage training are carried out. Joint constraints of offset regularization and modulation entropy regularization are introduced.
It significantly improves the model's feature extraction capabilities, enhances inference accuracy and generalization ability, strengthens the model's robustness, and enables it to better adapt to facial occlusion and pose changes.
Smart Images
Figure CN122049966B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of facial recognition technology in the field of image or video recognition or understanding, and specifically relates to an auxiliary detection method for facial depression based on dynamic deformable convolution. Background Technology
[0002] With the development of artificial intelligence technology, automated detection methods based on facial expressions and behavioral characteristics have gradually become an important research direction for the auxiliary diagnosis of depression.
[0003] The closest existing technology to this invention is the use of a deep learning-based facial recognition model for depression to assist in facial depression detection. Specifically, this includes Deformable Convolution and Dynamic Convolution. A typical example of Deformable Convolution is DCNv3 (Deformable Convolution v3), whose core goal is to model the offset positions and sampling weights of the convolution kernel using a simpler and more unified set of parameterizations while maintaining the convolution structure and computational friendliness. DCNv3 introduces learnable offsets and learnable weights for each convolution kernel point and divides the channels into multiple groups, with each group independently predicting its own offset and weights to reduce parameter computation. The key to DCNv3 is that it uses a lightweight convolution head to predict all offsets and weights from the input features at once, without requiring extra branches, and then appropriately scales or constrains the offsets and weights before finally performing sampling and convolution calculations. After sampling the input features by coordinates, the sampled features are weighted and summed with the modulation weights and convolution kernel parameters in groups to obtain the output channels. The core idea of dynamic convolution is that the weights of the convolution kernel are no longer fixed, but are adaptively generated or combined according to the input features. This allows the network to use different convolution kernels for different samples at the same location, improving the model's fitting ability. A classic implementation of dynamic convolution is multi-expert convolution kernels + routing weights. For a certain convolutional layer, dynamic convolution stores multiple sets of expert convolution kernels. Each expert convolution kernel has the same shape and dimensions as a regular convolution kernel. The dynamic convolution kernel is generated by weighting and combining these expert convolution kernels according to the input content. This is a typical implementation of dynamic convolution, conditional parameterized convolution, and other similar methods.
[0004] Deformable convolutions like DCNv3 only perform positional analysis and single-layer kernel sampling point weighting, with their dynamism primarily manifested in the spatial dimension. In the channel dimension, they still use shared kernel weights, lacking adaptation to different semantic patterns. This makes them insufficiently precise for scenarios requiring differentiated modeling of different channel subspaces (such as emotion and depression-related micro-expressions). Furthermore, DCNv3 typically learns its offsets and modulation parameters end-to-end without applying explicit regularization to parameters like offset magnitude and distribution, and weight entropy. This can lead to excessively large offset values that bounce around in the image background when the task complexity is high. Additionally, the weights may be too sharp (e.g., concentrated at a single kernel point) or too dispersed, resulting in unstable features.
[0005] Existing deep learning-based facial recognition models for depression typically rely on high-quality video datasets collected in laboratory settings for training. These models often require pre-defined facial information completeness during feature extraction. However, in real-world, non-laboratory scenarios, subjects' faces are frequently partially obscured due to head posture, jewelry, gestures, or environmental factors. Existing deep learning-based facial recognition models for depression exhibit significantly reduced feature extraction capabilities when faced with such facial occlusion or incompleteness, resulting in low inference accuracy, weak generalization ability, and insufficient robustness in practical deployments. Summary of the Invention
[0006] To address the shortcomings of existing deep learning-based facial recognition models for depression, such as weak feature extraction capabilities, low inference accuracy, weak generalization ability, and insufficient robustness, this invention provides a facial depression auxiliary detection method based on dynamic deformable convolution.
[0007] The technical solution adopted by this invention to solve the technical problem is as follows:
[0008] This invention provides an auxiliary detection method for facial depression based on dynamic deformable convolution, comprising the following steps:
[0009] S1: Facial video data acquisition and uniform sampling of frame sequences;
[0010] S2: Perform adaptive Gamma illumination correction and USM deblurring and sharpening processing on the video frames sampled in step S1;
[0011] S3: Generate RGB image frames and corresponding optical flow feature maps using the video frame images processed in step S2;
[0012] S4: Building an integrated HD 3The dual-stream DCNN network model with dynamic convolution module includes a spatial appearance stream backbone network and a temporal dynamic stream backbone network. The spatial appearance stream input tensor is obtained from RGB image frames and input to the spatial appearance stream backbone network to extract global feature vectors. Similarly, the temporal dynamic stream input tensor is obtained from optical flow feature maps and input to the temporal dynamic stream backbone network to extract global feature vectors. These global feature vectors are concatenated along the channel dimension to form a fused feature, which is then used for regression analysis to generate a predicted depression level. Five-fold cross-validation and a two-stage training strategy are implemented. The model is evaluated based on the predicted depression level and the visualization of the results. The best model is then selected for inference and analysis.
[0013] Furthermore, in step S3, the video frame image is cropped by facial key feature points based on the face recognition open-source library Dlib to obtain RGB image frames; the inter-frame pixel displacement and displacement intensity are calculated using the Farneback optical flow extraction algorithm, and the channels are normalized and stacked into an optical flow feature map.
[0014] Furthermore, in step S4, the method for obtaining the spatial appearance stream input tensor is as follows: converting RGB image frames into three-channel RGB images, performing random data augmentation on single-frame RGB images in training mode, canceling randomness in verification and testing modes, performing uniform scaling and center cropping, converting the augmented RGB images into tensor form and performing normalization processing to obtain the spatial appearance stream input tensor; the method for obtaining the temporal dynamic stream input tensor is as follows: reading the npy file storing optical flow and performing random data augmentation, encoding the optical flow information into three-channel image form, combining the obtained three-channel optical flow images into tensor form and performing normalization processing to obtain the temporal dynamic stream input tensor.
[0015] Furthermore, in step S4, the spatial appearance flow backbone network is based on the GoogLeNet model, retaining convolutional layers, pooling layers, and the Inception module, and introducing HD into the spatial appearance flow backbone network. 3 The dynamic convolution module, the HD 3 The dynamic convolution module introduces a dynamic deformable convolution mechanism with an expert library; the temporal dynamic flow backbone network adopts the same GoogLeNet model structure as the spatial appearance flow backbone network, but does not load pre-trained weights and starts training from random initialization.
[0016] Furthermore, in step S4, HD is enabled. 3 Dynamic convolution modules; convolution replacement only in the GoogLeNet model; patch replacement only for the inception5a and inception5b modules in the GoogLeNet model; replacement of convolutional layers with kernel size greater than or equal to 3; HD 3The dynamic convolution module is divided into G channel groups, and each channel group contains E expert convolution kernels; offset regularization coefficients and modulation entropy regularization coefficients are set.
[0017] Furthermore, in step S4, the HD 3 The dynamic convolution module employs group convolution and multi-expert convolution kernel mechanisms to achieve spatial adaptive convolution, generating dynamic convolution kernels corresponding to the current local features. It predicts the offset and weight modulation coefficients based on the current local features and performs softmax normalization on the weights of all convolution kernel points at each location. During forward propagation, the HD... 3 The dynamic convolution module superimposes the predicted offset and the geometric position of the convolution kernel, uses bilinear interpolation to obtain sampling features, multiplies them by the corresponding modulation weights, and then performs a linear combination with the dynamic convolution kernel. After concatenation, the final dynamic convolution result is obtained.
[0018] Furthermore, in step S4, a two-stream fusion regression head is used to concatenate the global feature vectors extracted by the spatial appearance stream backbone network and the temporal dynamic stream backbone network into a fusion feature in the channel dimension. The fusion feature is then input into a three-layer fully connected module: first through a fully connected layer and ReLU activation, then through Dropout regularization, and finally through a single-output neuron to output the fused depression level prediction value.
[0019] Furthermore, in step S4, K-fold cross-validation is performed, and five-fold cross-validation is performed on a video-by-video basis: the video index is divided into a training video set and a validation video set using a K-fold partitioner, the frame-optical flow plane index list is traversed, and the frame-optical flow pairs belonging to the training video are assigned to the training subset, and the frame-optical flow pairs belonging to the validation video are assigned to the validation subset; a training data loader and a validation data loader are constructed for each fold of data, and multi-threaded loading is performed after setting the batch size.
[0020] Furthermore, in step S4, the two-stage training strategy is as follows:
[0021] The Huber loss function was used as the regression loss, optimized using the AdamW optimizer and a cosine annealing learning rate scheduler. The Huber loss between the predicted depression level and the true label was used as the main loss, and HD was superimposed. 3 The regularization term constitutes the total training loss;
[0022] Phase 1 training: Initialize the dual-stream DCNN network model, freeze the backbone parameters of the spatial appearance stream backbone network, and participate in the training with the backbone parameters of the temporal dynamic stream backbone network.
[0023] Second stage training: Reconstruct the dual-stream DCNN network model on the same fold data, unfreeze the backbone parameters of the spatial appearance stream backbone network, and have both the spatial appearance stream backbone network and the temporal dynamic stream backbone network participate in end-to-end fine-tuning.
[0024] After each training round, video-level validation and early stopping control are performed on the validation set: the average of all frame-level predictions for the same video is taken, and the mean square error, mean absolute error, root mean square error, and coefficient of determination are calculated as evaluation indicators. The parameters of the model with the best performance in the current compromise validation are saved as a weight file.
[0025] Furthermore, in step S4, after completing the five-fold cross-validation, the mean absolute error of each fold at the end of the second stage of training is calculated, and the fold with the smallest error is selected as the best fold. A test set is constructed using the test partition for evaluation only, and the mean absolute error, root mean square error, and coefficient of determination are calculated on the test set as the final performance indicators of the model.
[0026] The beneficial effects of this invention are:
[0027] 1. Significantly enhanced feature extraction capability: This invention proposes a HD feature extraction method employing group convolution and multi-expert convolution kernel mechanisms. 3 The dynamic convolution module, based on grouping its input channels, configures an expert database for each channel group. The expert convolution kernel router adaptively combines them according to the input features, which not only retains the grouping advantages of DCNv3, but also introduces expert diversity. This significantly improves the convolution kernel's adaptability to different expressions, individual differences, and abnormal occlusion, and further enhances the model's feature extraction capabilities.
[0028] 2. High inference accuracy: This invention features an HD inference mechanism with a dynamic deformable convolutional structure and an expert database. 3 The dynamic convolution module improves the two-stream DCNN network model, making it perform better on the depression regression task in the video modality. Its completely decoupled feature can ensure the stability of model training, thereby improving the inference accuracy of the model.
[0029] 3. Strong generalization ability: This invention employs softmax normalization in deformable sampling and weight modulation, and introduces joint constraints of offset regularization and modulation entropy regularization. This controls the spatial range of deformable sampling while avoiding excessively extreme modulation weights, thus improving HD... 3 The dynamic convolution module exhibits better numerical stability and generalization ability in actual training.
[0030] 4. Enhanced robustness: This invention introduces a more robust HD layer into the constructed dual-stream DCNN network model. 3 The dynamic convolution module enables the model to make fuller use of spatiotemporal information and achieve stronger physical consistency. Among these features, the introduced HD...3 The dynamic convolution module boasts strong expressive power and efficient parameters. Employing group convolution and multi-expert kernel mechanisms to achieve spatially adaptive convolution, it can more precisely track local deformations and pose changes. Furthermore, this invention does not simply introduce a DCN layer outside the GoogLeNet model; instead, it embeds the Inception module within the GoogLeNet model. Compared to simply stacking DCN layers before and after the backbone network, this more refined replacement method maximizes the multi-scale characteristics of the Inception module's big.mega-kernel convolution without compromising the overall network optimization properties, thereby improving model robustness. Attached Figure Description
[0031] Figure 1 This is a schematic diagram of the overall process of an auxiliary detection method for facial depression based on dynamic deformable convolution provided by the present invention.
[0032] Figure 2 This is a diagram of the dual-stream DCNN network model constructed in this invention. Detailed Implementation
[0033] The present invention will be further described in detail below with reference to the accompanying drawings.
[0034] like Figure 1 As shown, the present invention provides an auxiliary detection method for facial depression based on dynamic deformable convolution, the specific implementation process of which is as follows:
[0035] Step S1: Facial video data acquisition and uniform sampling of frame sequence;
[0036] Step S1.1: Obtain raw video data, which can be publicly available facial video or image modality datasets of depression, or self-collected high-definition video data, ensuring that the video resolution meets the minimum input requirements (e.g., 224 × 224).
[0037] Step S1.2: Perform frame extraction on the continuous video stream. Specifically, first extract all frames of a video segment, remove frames where no key facial feature points can be detected, and then uniformly sample the frame sequence in chronological order to ensure the temporal order of the video frames. At least 80 frames are sampled for each video of the subject to ensure that there are no invalid frames containing severe noise.
[0038] Step S2: Image enhancement preprocessing, including adaptive Gamma illumination correction and USM deblurring and sharpening;
[0039] Step S2.1: Adaptive Gamma illumination correction;
[0040] Adaptive Gamma illumination normalization is performed on the sampled video frames. Based on the brightness histogram distribution of the video frame images, the brightness distribution of the video frame images is automatically adjusted through a non-linear mapping curve to solve the problem of uneven illumination in the video and restore details in dark areas.
[0041] Step S2.2: Unsharp Mask (USM) deblurring and sharpening process;
[0042] Unsharp Masking (USM) is applied to the video frame images after adaptive gamma illumination correction to remove blur and sharpen them. By extracting the high-frequency components of the video frame images and superimposing them back into the original video frame images, the contrast of object edges is enhanced, the unclear boundaries caused by motion blur are eliminated, and the contours of the motion are made clearer.
[0043] Step S3: Generate RGB image frames and corresponding optical flow feature maps;
[0044] Step S3.1: Prepare the input data for the spatial appearance stream;
[0045] The video frame image processed in step S2 is cropped using the face recognition open-source library Dlib to obtain RGB image frames by cropping key facial feature points. Useless features in the video frame image are removed, and the cropped face image is uniformly scaled to 224 × 224 pixels.
[0046] Step S3.2: Prepare the input data for the dynamic stream of the time;
[0047] The Farneback optical flow extraction algorithm is used to calculate the inter-frame pixel displacement (u,v) and displacement intensity, normalize the channels to the range of 0-255, and stack them into an optical flow feature map to capture the dynamic motion features of the subject.
[0048] Step S4: Construct a dual-stream DCNN network model, perform feature fusion and training; evaluate the model based on the output results and result visualization, and select the best model for subsequent inference, data analysis and other tasks.
[0049] Step S4.1: Build integrated HD 3 A two-stream DCNN network model with dynamic convolution modules;
[0050] Step S4.1.1: First, establish a mapping relationship between each subject's video and its label and the .npy file storing optical flow, and construct a complete subject sample of video-optical flow-depression rating label, ensuring that each .npy file storing optical flow can match an unprocessed original frame from step S1, and finally discard an extra frame.
[0051] Step S4.1.2: Obtain the spatial appearance stream input tensor;
[0052] Read the RGB image frame generated in step S3.1 and convert it into a three-channel RGB image. In training mode, perform consistent random data augmentation on all single-frame RGB images, including scaling the RGB image proportionally to 256 × 256 pixels, performing a horizontal random flip with a 50% probability, randomly rotating the image, perturbing the image with color dithering, and randomly cropping a 224 × 224 region on the scaled image. In validation and testing mode, cancel the randomness and only perform uniform scaling and center cropping on the RGB image to obtain an image of size 224 × 224. Convert the augmented RGB image into tensor form (channel order is 3 × 224 × 224) and standardize it according to the statistics of the ImageNet dataset, subtracting the corresponding mean from each channel and then dividing by the standard deviation to obtain the spatial appearance stream input tensor.
[0053] Step S4.1.3: Obtain the time dynamic stream input tensor;
[0054] Read the .npy file storing the optical flow from the optical flow directory. This file records the horizontal and vertical displacement fields between two consecutive frames. The original shape of the optical flow feature map obtained in step S3.2 is 2 × H × W (H is the height, W is the width). First, perform the same spatial transformation processing as the random data augmentation processing in step S4.1.2, and then encode the optical flow information into a three-channel image form: the first channel is the horizontal component u, which is truncated by a symmetrical range and mapped to [0, 255]; the second channel is the vertical component v, which is also truncated by the center and mapped to [0, 255]; the third channel is the optical flow vector magnitude. The optical flow images are then normalized to [0, 255]. The three-channel optical flow images are combined into a 3×224×224 tensor, and then normalized according to the ImageNet mean and standard deviation to obtain the temporal dynamic flow input tensor.
[0055] Step S4.1.4: Construct the spatial appearance flow backbone network and the temporal dynamic flow backbone network;
[0056] A spatial appearance flow backbone network is constructed based on the GoogLeNet model provided by the torchvision visual model library, retaining the structure of convolutional layers, pooling layers, and the Inception module, from input to average pooling layer. During the training phase of the spatial appearance flow, weights pre-trained on the ImageNet dataset are loaded to learn low-level features. In the forward propagation, the spatial appearance flow input tensor (3 × 224 × 224) obtained in step S4.1.2 is processed through a series of convolutions, max pooling, and Inception modules, and then global average pooling is used to obtain a global feature vector of dimension 1024.
[0057] A temporal dynamic flow backbone network is constructed, using the same GoogLeNet model structure as the spatial appearance flow backbone network, but without loading pre-trained weights, and training starts from random initialization; the temporal dynamic flow input tensor (size 3 × 224 × 224) obtained in step S4.1.2 is passed through the temporal dynamic flow backbone network and outputs a global feature vector with a dimension of 1024 after global average pooling, which is used to describe the inter-frame motion pattern.
[0058] Step S4.1.5: Introduce HD into the spatial appearance flow backbone network 3 The dynamic convolution module, this HD 3 The dynamic convolution module introduces a dynamic deformable convolution mechanism with an expert library, which can adaptively extract spatial features and cope with possible facial occlusion.
[0059] Specifically, HD is set through a unified configuration class. 3 The structural parameters and regularization coefficients of the dynamic convolution module, where HD is enabled. 3 Dynamic convolution modules; convolution replacement only in the GoogLeNet model; patch replacement only for the inception5a and inception5b Inception modules in the GoogLeNet model; replacement of convolutional layers with kernel sizes greater than or equal to 3; HD 3 The dynamic convolution module is divided into G = 4 channel groups, and each channel group contains E = 4 expert convolution kernels; the offset regularization coefficient and the modulation entropy regularization coefficient are set to... and For each selected Inception module's base convolutional unit, the kernel size of its internal convolutional layers is detected. When the kernel size is greater than or equal to 3, a custom HD is used. 3 The dynamic convolution module replaces its original two-dimensional convolutional layer, where the two-dimensional convolutional layer refers to the 3 × 3 or 5 × 5 convolutional kernel in the GoogLeNet model.
[0060] HD 3 The specific implementation process of the dynamic convolution module is as follows:
[0061] (1) Spatial adaptive convolution to obtain dynamic convolution kernels;
[0062] In HD 3 In the dynamic convolution module, spatial adaptive convolution is achieved using group convolution and a multi-expert convolution kernel mechanism: the input channels are evenly divided into groups G, and each group of channels shares a set of expert convolution kernel parameters. An expert library is configured for each channel group, with a shape approximately [G, E, Og, Cg, K], where Cg and Og are the number of input and output channels in each group, respectively, and K is the square of the spatial convolution kernel size. An expert convolution kernel router adaptively generates weight coefficients for each group of channels on E expert convolution kernels based on the current local features. The combined weights of each expert group are obtained through softmax, and then a weighted sum is performed on each group of expert convolution kernels to generate the dynamic convolution kernel corresponding to the current local features.
[0063] (2) Deformable sampling and weighted modulation;
[0064] To achieve deformable sampling and weight modulation, HD 3 The dynamic convolution module further predicts the offset and weight modulation coefficients based on the current local features: the offset predictor receives the local feature map and outputs the two-dimensional offset (dy, dx) corresponding to each channel group and each convolution kernel point (K positions in total). The two-dimensional offset (dy, dx) is scaled and limited to a set pixel range by the hyperbolic tangent function to prevent the offset from being too large. At the same time, normalized modulation weights are generated, and the weights of all convolution kernel points are normalized by softmax at each position to achieve adaptive adjustment of the contribution of different convolution kernel points.
[0065] (3) In forward propagation, HD 3 The dynamic convolution module calculates the spatial dimensions of the feature map based on the stride and padding, constructs a corresponding convolution sampling reference grid for each output position, and superimposes the predicted offset and the geometric position of the convolution kernel. It uses bilinear interpolation to sample K features from the feature map according to the above two-dimensional offset (dy, dx), and multiplies them by the corresponding modulation weights. The sampled features are then linearly combined with the dynamic convolution kernel obtained based on the expert convolution kernel router through tensor multiplication to obtain the output features in each channel group. Finally, the output features of each channel group are concatenated in the channel dimension to obtain the final dynamic convolution result.
[0066] (4) During training mode, regularization constraints are applied to the sum of squared offsets and the entropy term of the modulation weights, respectively, to form additional HD.3 Regularization loss is used to suppress excessive offset and overly sharp modulation, thereby stabilizing the training process.
[0067] Specifically, such as Figure 2 As shown, the spatial appearance flow input tensor is fed into the spatial appearance flow backbone network, and then sequentially passes through layers 1-4 of the GoogLeNet model's convolutional layers and HD. 3 The spatial flow features, i.e., the 1024-dimensional global feature vector, are obtained through dynamic convolutional modules, average pooling layers, and flattening. The temporal dynamic flow input tensor is input into the temporal dynamic flow backbone network, and then passes through the first two convolutional layers, convolutional layer 3a, convolutional layer 3b, max pooling layer, convolutional layers 4a-4e, max pooling layer, convolutional layer 5a, convolutional layer 5b, average pooling layer, and flattening of the GoogLeNet model to obtain the temporal flow features, i.e., the 1024-dimensional global feature vector. The obtained spatial flow features and temporal flow features are used for feature concatenation and fusion.
[0068] Step S4.2: Two-stream feature splicing, fusion, and regression;
[0069] A two-stream fusion regression head is used. First, the 1024-dimensional global feature vectors extracted by the spatial appearance stream backbone network and the temporal dynamic stream backbone network are concatenated into a 2048-dimensional fusion feature in the channel dimension. Then, the 2048-dimensional fusion feature is input into a three-layer fully connected module: first, it is passed through a 512-dimensional fully connected layer and ReLU activation, then regularized by Dropout (probability 0.5), and finally the fused depression level prediction value is output through a single output neuron.
[0070] Step S4.3: Perform K-fold cross-validation partitioning, using video units rather than frame units for five-fold cross-validation;
[0071] A K-fold partitioner is used to divide the video index into training and validation video sets. For each video set, the frame-optical flow plane index list is traversed, and frame-optical flow pairs belonging to training videos are assigned to the training subset, while those belonging to validation videos are assigned to the validation subset. A training data loader and a validation data loader are constructed for each fold of data, using a batch size of 32 and multi-threaded loading.
[0072] Step S4.4: Two-stage training strategy;
[0073] The Huber loss function was used as the regression loss, with a smoothing threshold of 7.0. Optimization was performed using the AdamW optimizer and a cosine annealing learning rate scheduler: the first-stage learning rate was set to... The learning rate for the second stage is set to The weight decay coefficient is uniformly set to .
[0074] Phase 1 Training: Initialize the dual-stream DCNN network model, where the backbone parameters of the spatial appearance stream backbone network are frozen, while the backbone parameters of the temporal dynamic stream backbone network are used in training; based on the 1024-dimensional global feature vectors extracted from the spatial appearance stream and temporal dynamic stream backbone networks, a depression level prediction value is output through a two-stream fusion regression head; the Huber loss between the depression level prediction value and the true label is used as the main loss, and HD is superimposed. 3 The regularization term constitutes the total training loss.
[0075] After each training epoch, evaluation is performed on the validation set at the video level: the mean squared error, mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²) are calculated by averaging all frame-level predictions for the same video. 2 As an evaluation metric, the number of early stopping rounds is set to 10, and the parameters of the best-performing model in the current compromise validation are saved as a weight file.
[0076] Second-stage training: Reconstruct the two-stream DCNN network model on the same folded data and restore the HD exactly the same as the first-stage training. 3 Configure and unfreeze the backbone parameters of the spatial appearance flow backbone network, allowing both the spatial appearance flow and temporal dynamic flow branches to participate in end-to-end fine-tuning. Further train the model with a smaller learning rate, still using the Huber loss between the depression level predictions output by the two-stream fusion regression head and the true labels, superimposed with HD. 3 The regularization term is used as the total training loss. Video-level validation and early stopping control are performed in the same way as in the first stage of training. The parameters of the model with the best performance in the current compromise validation are saved as a weight file.
[0077] Step S4.5: Recording and visualizing the training process;
[0078] In the two-stage training, the training loss, validation loss, and mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²) of the fused regression output are recorded for each epoch. 2 The system generates corresponding training curves based on the indicators and saves them to disk. For the best model at each stage, scatter plots and error histograms are used to display the relationship between the predicted depression level and the true label, as well as the residual distribution and other regression results, to facilitate subsequent analysis of model performance.
[0079] Step S4.6: Independent test set evaluation and result output;
[0080] After completing the five-fold cross-validation, the mean absolute error (MAE) of each fold at the end of the second phase of training is calculated, and the fold with the smallest MAE is selected as the optimal fold. A test set for evaluation is constructed using the test partition, with all random data augmentation disabled. Fixed scaling and center cropping are performed only on video frames and optical flow. Then, the spatial appearance flow input tensor and temporal dynamic flow input tensor are generated according to the optical flow graph augmentation procedure (normalization) in step S4.1.3. The mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²) are calculated on the test set. 2 ), which serves as the final performance metric for the model.
[0081] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for auxiliary detection of facial depression based on dynamic deformable convolution, characterized in that, Includes the following steps: S1: Facial video data acquisition and uniform sampling of frame sequences; S2: Perform adaptive Gamma illumination correction and USM deblurring and sharpening processing on the video frames sampled in step S1; S3: Generate RGB image frames and corresponding optical flow feature maps using the video frame images processed in step S2; S4: Building an integrated HD 3 A dual-stream DCNN network model with dynamic convolution module includes a spatial appearance stream backbone network and a temporal dynamic stream backbone network. The spatial appearance stream input tensor is obtained from RGB image frames and input to the spatial appearance stream backbone network to extract global feature vectors. Similarly, the temporal dynamic stream input tensor is obtained from optical flow feature maps and input to the temporal dynamic stream backbone network to extract global feature vectors. These global feature vectors are concatenated along the channel dimension to form a fused feature, which is then used for regression analysis to generate a predicted depression level. Five-fold cross-validation and a two-stage training strategy are implemented. The model is evaluated based on the predicted depression level and the visualization of the results. The best model is selected for inference and analysis. The spatial appearance flow backbone network is based on the GoogLeNet model, retaining convolutional layers, pooling layers, and the Inception module, and introducing HD into the spatial appearance flow backbone network. 3 The dynamic convolution module, the HD 3 The dynamic convolution module introduces a dynamic deformable convolution mechanism with an expert library; the temporal dynamic flow backbone network adopts the same GoogLeNet model structure as the spatial appearance flow backbone network, but does not load pre-trained weights and starts training from random initialization. The HD 3 The dynamic convolution module employs group convolution and multi-expert convolution kernel mechanisms to achieve spatial adaptive convolution, generating dynamic convolution kernels corresponding to the current local features, and predicting offsets and weight modulation coefficients based on the current local features; during forward propagation, the HD... 3 The dynamic convolution module superimposes the predicted offset and the geometric position of the convolution kernel, uses bilinear interpolation to obtain sampling features, multiplies them by the corresponding modulation weights, and then performs a linear combination with the dynamic convolution kernel. After concatenation, the final dynamic convolution result is obtained.
2. The method for auxiliary detection of facial depression based on dynamic deformable convolution according to claim 1, characterized in that, In step S3, the video frame image is cropped by facial key feature points based on the face recognition open-source library Dlib to obtain RGB image frames; the inter-frame pixel displacement and displacement intensity are calculated using the Farneback optical flow extraction algorithm, the channels are normalized and stacked into an optical flow feature map.
3. The facial depression auxiliary detection method based on dynamic deformable convolution according to claim 1, characterized in that, In step S4, the method for obtaining the spatial appearance stream input tensor is as follows: convert the RGB image frame into a three-channel RGB image; perform random data augmentation on a single-frame RGB image in training mode; cancel randomness in verification and testing modes; perform uniform scaling and center cropping; convert the augmented RGB image into tensor form and perform normalization processing to obtain the spatial appearance stream input tensor; the method for obtaining the temporal dynamic stream input tensor is as follows: read the npy file storing the optical flow and perform random data augmentation; encode the optical flow information into a three-channel image form; combine the obtained three-channel optical flow images into a tensor form and perform normalization processing to obtain the temporal dynamic stream input tensor.
4. The method for auxiliary detection of facial depression based on dynamic deformable convolution according to claim 1, characterized in that, In step S4, a two-stream fusion regression head is used to concatenate the global feature vectors extracted by the spatial appearance stream backbone network and the temporal dynamic stream backbone network into a fusion feature in the channel dimension. The fusion feature is then input into a three-layer fully connected module: first, it is activated by a fully connected layer and ReLU, then it is regularized by Dropout, and finally the fused depression level prediction value is output through a single output neuron.
5. The method for auxiliary detection of facial depression based on dynamic deformable convolution according to claim 1, characterized in that, In step S4, K-fold cross-validation is performed, and five-fold cross-validation is performed on a video-by-video basis: the video index is divided into training video set and validation video set using the K-fold partitioner, the frame-optical flow plane index list is traversed, and the frame-optical flow pairs belonging to the training video are assigned to the training subset, and the frame-optical flow pairs belonging to the validation video are assigned to the validation subset; a training data loader and a validation data loader are built for each fold of data, and multi-threaded loading is performed after setting the batch size.
6. The method for auxiliary detection of facial depression based on dynamic deformable convolution according to claim 1, characterized in that, In step S4, the two-stage training strategy is as follows: The Huber loss function was used as the regression loss, optimized using the AdamW optimizer and a cosine annealing learning rate scheduler. The Huber loss between the predicted depression level and the true label was used as the main loss, and HD was superimposed. 3 The regularization term constitutes the total training loss; Phase 1 training: Initialize the dual-stream DCNN network model, freeze the backbone parameters of the spatial appearance stream backbone network, and participate in the training with the backbone parameters of the temporal dynamic stream backbone network. Second stage training: Reconstruct the dual-stream DCNN network model on the same fold data, unfreeze the backbone parameters of the spatial appearance stream backbone network, and have both the spatial appearance stream backbone network and the temporal dynamic stream backbone network participate in end-to-end fine-tuning. After each training round, video-level validation and early stopping control are performed on the validation set: the average of all frame-level predictions for the same video is taken, and the mean square error, mean absolute error, root mean square error, and coefficient of determination are calculated as evaluation indicators. The parameters of the model with the best performance in the current compromise validation are saved as a weight file.
7. The method for auxiliary detection of facial depression based on dynamic deformable convolution according to claim 1, characterized in that, In step S4, after completing the five-fold cross-validation, the mean absolute error of each fold at the end of the second stage of training is calculated, and the fold with the smallest error is selected as the best fold. A test set is constructed using the test partition for evaluation only, and the mean absolute error, root mean square error, and coefficient of determination are calculated on the test set as the final performance indicators of the model.