Depression recognition method and system based on double-branch multi-scale 3D coordinate attention

Abstract

The application provides a depression recognition method and system based on a double-branch multi-scale 3D coordinate attention, belongs to the technical field of deep learning and the technical field of medical image processing, and comprises a deep learning network module which is configured to: construct a double-branch multi-scale 3D coordinate attention deep learning network, wherein the network comprises a 3D convolution layer and a max pooling layer, is used for extracting basic space-time features of an input face standardized face video, and outputs initial face features; a main feature branch which takes a 3D residual network as a backbone and embeds a multi-scale 3D coordinate attention module, is used for extracting multi-receptive field face static features with space-time position perception from features output by a mask layer; an inter-frame difference dynamic branch which is used for extracting face micro-expression dynamic features from features output by the mask layer through time domain difference operation and scale calibration, and outputs dynamic features matched in size with the main feature branch; and an output depression prediction result.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning technology and medical image processing technology, and particularly relates to a method and system for depression recognition based on bi-branch multi-scale 3D coordinate attention. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the development of deep learning technology, computer vision technology has been gradually applied to the field of depression recognition. Because deep learning models can automatically learn more discriminative multi-level features directly from original images or video sequences, significantly reducing the reliance on manual feature design and domain prior knowledge, deep learning-based methods for depression recognition have become one of the mainstream technologies.

[0004] Current deep learning-based methods for facial depression recognition still face several challenges. For instance, most models employ a simplistic approach to facial feature modeling, making it difficult to accurately pinpoint facial regions and key time segments highly correlated with depression. Furthermore, convolutional structures with fixed receptive fields cannot adapt to multi-scale dynamic changes in the face. Additionally, most models do not adequately consider facial dynamic information, resulting in insufficient extraction of key depression-related features and consequently, lower accuracy and interpretability in depression recognition. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a depression recognition method and system based on bi-branch multi-scale 3D coordinate attention, which can accurately locate facial features related to depression recognition, while adapting to multi-scale dynamic changes of the face and extracting more facial dynamic information, thereby improving the accuracy and interpretability of depression recognition.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: Firstly, a depression recognition system based on bi-branch multi-scale 3D coordinate attention is disclosed, including: The facial video acquisition and data preprocessing module is configured to: acquire facial videos and perform face standardization through data preprocessing, and output face-standardized facial videos; The deep learning network module is configured to: construct a dual-branch, multi-scale 3D coordinate attention deep learning network, the network comprising: 3D convolutional layers and max pooling layers are used to extract basic spatiotemporal features from the input face-standardized facial video and output initial facial features; The masking layer is used to suppress background noise and enhance key facial features of the initial facial features, and to generate dual-branch common input features. The main feature branch uses a 3D residual network as its backbone and embeds a multi-scale 3D coordinate attention module to extract multi-receptive field, spatiotemporally aware facial static features from the mask layer output. The inter-frame difference dynamic branch extracts facial micro-expression dynamic features from the features output by the mask layer through temporal difference operations and scale calibration, and outputs dynamic features that match the size of the main feature branch. The concatenation layer concatenates the features of the two branches along the channel dimension, performs feature fusion through a convolutional layer, and then maps them through an adaptive pooling layer and a fully connected layer to output the depression prediction result.

[0007] As a further technical solution, it also includes: a training optimization step, which uses a weighted fusion of mean squared error loss and smoothed L1 loss as the total loss function, and uses mean absolute error and root mean square error as performance evaluation indicators.

[0008] As a further technical solution, when suppressing background noise and enhancing key facial features of the initial facial features, the following are included: For a given feature, average pooling and max pooling are first performed along the channel dimension to extract global background and salient contour features in the spatial dimension; A residual connection mechanism is adopted: the given features are weighted and filtered using a mask, and then the weighted result is added back to the given features.

[0009] As a further technical solution, the input features are mapped in the main feature branch. Then, the concatenated features are fused and compressed through the dimension control layer. After passing through the coordinate attention module, the multi-scale feature map is reweighted pixel by pixel using the obtained three orthogonal axis attention weights.

[0010] As a further technical solution, an adaptive calibration mechanism is introduced at the end of the inter-frame differential dynamic branch.

[0011] As a further technical solution, the process of standardizing faces through data preprocessing includes: Acquire facial interview videos to ensure the clarity of facial features and the completeness of dynamic information; The frame extraction unit decodes the video frame by frame, breaking the continuous video into a sequence of single-frame images. At the same time, the image sequence is initially screened to remove invalid frames, blurry frames, and black screen frames. The face processing unit uses a face detector and a facial key point predictor to perform face region localization and pose alignment on a single frame image. The aligned facial images are uniformly cropped and scaled to achieve spatial standardization; For scenarios where face detection fails, a fault-tolerant cropping mechanism is configured to adaptively crop the facial region through a preset strategy of rightward tilting of the face center and expanding the cropping area.

[0012] As a further technical solution, face region localization and pose alignment are performed on a single frame image: the face region in the image is located by a face detector, and the facial key point predictor is used to obtain the facial contour and the coordinates of key feature points of the facial features. Facial pose correction is then completed based on the coordinates of the key feature points.

[0013] Secondly, a depression identification method based on bi-branch multi-scale 3D coordinate attention is disclosed, including: Acquire facial videos and perform face standardization through data preprocessing, then output standardized facial videos; Construct a dual-branch, multi-scale 3D coordinate attention deep learning network, the network comprising: 3D convolutional layers and max pooling layers are used to extract basic spatiotemporal features from the input face-standardized facial video and output initial facial features; The masking layer is used to suppress background noise and enhance key facial features of the initial facial features, and to generate dual-branch common input features. The main feature branch uses a 3D residual network as its backbone and embeds a multi-scale 3D coordinate attention module to extract multi-receptive field, spatiotemporally aware facial static features from the mask layer output. The inter-frame difference dynamic branch extracts facial micro-expression dynamic features from the features output by the mask layer through temporal difference operations and scale calibration, and outputs dynamic features that match the size of the main feature branch. The concatenation layer concatenates the features of the two branches along the channel dimension, performs feature fusion through a convolutional layer, and then maps them through an adaptive pooling layer and a fully connected layer to output the depression prediction result.

[0014] The above one or more technical solutions have the following beneficial effects: The technical solution of this invention takes facial video as input, completes face standardization through data preprocessing, and then inputs the video sequence into the network. It adopts a mode of extracting multi-scale static features with spatiotemporal coordinate awareness by backbone branch and extracting dynamic facial features by parallel inter-frame difference branch. Combined with facial guided mask to suppress background interference, the end-to-end regression assessment of depression level is achieved through dual-branch feature fusion, which can significantly improve the accuracy of depression recognition and model interpretability.

[0015] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0017] Figure 1 This is a structural diagram of a depression recognition method based on bi-branch multi-scale 3D coordinate attention according to this application; Figure 2 The flowchart below shows a depression recognition method based on bi-branch multi-scale 3D coordinate attention according to this application. Figure 3 This is a diagram of the dual-branch multi-scale 3D coordinate attention deep learning network structure proposed in this invention application; Figure 4 This is a diagram of the multi-scale 3D coordinate attention branch network structure according to this application; Figure 5 Here is a diagram of the differential branch network structure according to this application; Figure 6 This is a schematic diagram of the key points of the dlib facial cropping method in this invention application. Detailed Implementation

[0018] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0020] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0021] Example 1 To address the shortcomings of existing facial video-based depression recognition technologies, such as insufficient multi-scale spatiotemporal coordinate correlation modeling capabilities, poor fusion of static appearance and dynamic micro-expression features, inaccurate localization of key facial regions, and weak interpretability of model decisions, a depression recognition scheme is needed that can accurately capture depression-related facial features, adapt to multi-scale dynamic changes, extract key dynamic information, and possess high recognition accuracy and strong interpretability. This embodiment discloses a depression recognition method based on bi-branch multi-scale 3D coordinate attention, including: Acquire facial videos and perform face standardization through data preprocessing, then output standardized facial videos; Construct a dual-branch, multi-scale 3D coordinate attention deep learning network, the network comprising: 3D convolutional layers and max pooling layers are used to extract basic spatiotemporal features from the input face-standardized facial video and output initial facial features; The masking layer is used to suppress background noise and enhance key facial features of the initial facial features, and to generate dual-branch common input features. The main feature branch uses a 3D residual network as its backbone and embeds a multi-scale 3D coordinate attention module to extract multi-receptive field, spatiotemporally and spatially aware facial multi-scale static features from the mask layer output. The inter-frame difference dynamic branch extracts facial micro-expression dynamic features from the features output by the mask layer through temporal difference operations and scale calibration between adjacent frames, and outputs dynamic features that match the size of the main feature branch. The concatenation layer concatenates the features of the two branches along the channel dimension, performs feature fusion through a convolutional layer, and then maps them through an adaptive pooling layer and a fully connected layer to output the depression prediction result.

[0022] In one implementation example, the process of completing face standardization through data preprocessing includes: Acquire facial interview videos to ensure the clarity of facial features and the completeness of dynamic information; The frame extraction unit decodes the video frame by frame, breaking the continuous video into a sequence of single-frame images and automatically creating an output directory for file storage. At the same time, the image sequence is initially filtered to remove invalid frames, blurry frames, and black screen frames. The face processing unit uses a face detector and a facial key point predictor to perform face region localization and pose alignment on a single frame image. The aligned facial images are uniformly cropped and scaled to standardize the spatial dimensions and generate a standardized facial video sequence. For scenarios where face detection fails, a fault-tolerant cropping mechanism is configured to adaptively crop the facial region through a preset strategy of rightward tilting of the face center and expanding the cropping area.

[0023] In one implementation example, face region localization and pose alignment are performed on a single frame image: the face region in the image is located by a face detector, and the facial key point predictor is used to obtain the facial contour and the coordinates of key feature points of the facial features. Facial pose correction is then completed based on the coordinates of the key feature points.

[0024] In one implementation example, it also includes: a training optimization step, which uses a weighted fusion of mean squared error loss and smoothed L1 loss as the total loss function, and uses mean absolute error and root mean square error as performance evaluation metrics.

[0025] In one implementation example, when suppressing background noise and enhancing key facial features of the initial facial features, the process includes: For a given feature, average pooling and max pooling are first performed along the channel dimension to extract global background and salient contour features in the spatial dimension; A residual connection mechanism is adopted: the given features are weighted and filtered using a mask, and then the weighted result is added back to the given features.

[0026] In one implementation example, the input features are mapped in the main feature branch. Then, the concatenated features are fused and compressed through the dimension control layer. After passing through the coordinate attention module, the multi-scale feature map is reweighted pixel by pixel using the obtained three orthogonal axis attention weights.

[0027] In one implementation example, an adaptive calibration mechanism is introduced at the end of the inter-frame differential dynamic branch.

[0028] See appendix Figure 1 As shown, in actual recognition and prediction operation, the preprocessed facial image sequence is constructed as a temporal input, and 64 consecutive frames are randomly selected to form an input block. The input feature dimension satisfies Data augmentation was performed using methods such as random brightness adjustment, random sequence flipping, random horizontal flipping, and brightness and contrast adjustment. For batch size, For the number of channels, For timing length, This refers to the image size.

[0029] Forward propagation first performs basic feature extraction, converting the input tensor into a single vector. The process involves sequentially applying 3D convolution, batch normalization, ReLU activation, and max pooling, ultimately resulting in the following output:

[0030] Where conv_1 represents 3D convolution operation, bn1 represents batch normalization operation, relu represents ReLU non-linear activation operation, and maxpool represents max pooling operation. This represents the shallow features extracted by Xbase.

[0031] The basic features are input into the facial guidance mask module to complete feature enhancement, resulting in bifurcation features, which are denoted as... As the common input of the dual-branch structure, face_mask represents the face masking operation, and Xfork represents the enhanced features output after the masking operation.

[0032] Main branch road A The high-dimensional spatiotemporal features are extracted sequentially through a four-layer residual network. Its shape satisfies Layer_1 represents the first layer of 3D coordinate attention feature extraction, layer_2 represents the second layer, layer_3 represents the third layer, and layer_4 represents the fourth layer. All four layers are innovative networks based on residual networks. Xa represents the features of the main branch after these four layers of feature extraction.

[0033] The constraints on shape satisfaction are based on the consideration that ordinary 3D convolution lacks positional differentiation, and 2D attention loses temporal information, making it difficult to locate depression-related facial features and temporal segments. By applying these constraints, we can simultaneously model height, width, and temporal 3D coordinates, adaptively enhancing key areas such as the eyebrows, eyes, and mouth; this approach increases the number of parameters, maintains lightweight design, facilitates heatmap visualization, and improves model interpretability.

[0034] Inter-frame difference branch B As input, with Temporal spatial size To calibrate the target, T represents the time series length, H' and W' represent the height and width of a single-frame face image after feature extraction, and the output dynamic features are: And satisfy T represents the time sequence length, and H' and W' represent the height and width of a single-frame face image after feature extraction. X fork This represents the enhanced feature output after the masking operation. diff This indicates a difference extraction operation, which involves subtracting the features of the previous frame from the features of the next frame. X b This indicates the output characteristics of the differential branch.

[0035] The above conditions are imposed to account for the difficulty in capturing subtle facial dynamics associated with depression using a single-branch static convolution. These conditions allow for the explicit extraction of inter-frame micro-expression changes, complementing the main static features.

[0036] The dual-branch features are fused and mapped by first concatenating them along the channel dimension to obtain the features before fusion. `cat` represents the feature concatenation operation, where [Xa, Xb] represents the main static features and the difference dynamic features to be concatenated; `dim` represents the channel dimension. `Xcat` represents the concatenated features, which are then processed... Convolution completes the integration to obtain `feature_fusion_conv` represents a 1×1×1 3D convolutional fusion operation. `Xfusion` represents the fused features. Finally, global average pooling, flattening, Dropout, and fully connected mapping are performed sequentially to output a depression prediction score.

[0037] Here, avgpool represents average pooling, view represents tensor flattening, dropout represents regularization (which deactivates random neurons), fc represents a fully connected mapping layer, and Y represents the final predicted depression score.

[0038] The model training module uses a weighted fusion loss function to optimize the network. The total loss is calculated using the following formula:

[0039] in For loss weighting coefficients, For mean square error loss, To smooth out the $L_1$ loss.

[0040] Model training with mean absolute error and root mean square error The calculation formulas for these performance evaluation metrics are as follows: and .in For the sample size, To predict the score, This is a genuine BDI-II label.

[0041] The model training module uses the Adam optimizer with an initial learning rate of 0.001 and executes a gradient accumulation strategy, updating parameters every 10 gradient steps. Simultaneously, a cosine annealing restart mechanism is used to schedule the learning rate, with the current learning rate set at [value missing]. The calculation formula is:

[0042] in, For the current round within the cycle, This represents the total number of annealing cycles.

[0043] The training module has a fixed global random seed of 42, a maximum training epoch of 200 epochs, a Dropout layer is set before the fully connected layer to suppress overfitting, and the optimal model is selected based on the lowest error on the validation set.

[0044] In the model validation and testing phase, the model that performs best on the validation set is selected to predict K subsequences of 64 frames each, uniformly extracted from the long video. No data augmentation methods are used in this phase, and the network model is frozen. The final score is calculated by regression prediction, taking the average of all subsequence predictions.

[0045] Where K is the number of subsequences. This is the predicted value for a single subsequence.

[0046] Example 2 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0047] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0048] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0049] Example 4 The purpose of this embodiment is to provide a depression recognition system based on bi-branch multi-scale 3D coordinate attention, including: The facial video acquisition and data preprocessing module is configured to: acquire facial videos and perform face standardization through data preprocessing, and output face-standardized facial videos; The deep learning network module is configured to: construct a dual-branch, multi-scale 3D coordinate attention deep learning network, the network comprising: 3D convolutional layers and max pooling layers are used to extract basic spatiotemporal features from the input face-standardized facial video and output initial facial features; The masking layer is used to suppress background noise and enhance key facial features of the initial facial features, and to generate dual-branch common input features. The main feature branch uses a 3D residual network as its backbone and embeds a multi-scale 3D coordinate attention module to extract multi-receptive field, spatiotemporally aware facial static features from the mask layer output. The inter-frame difference dynamic branch extracts facial micro-expression dynamic features from the features output by the mask layer through temporal difference operations and scale calibration, and outputs dynamic features that match the size of the main feature branch. The concatenation layer concatenates the features of the two branches along the channel dimension, performs feature fusion through a convolutional layer, and then maps them through an adaptive pooling layer and a fully connected layer to output the depression prediction result.

[0050] Regarding the facial video acquisition and data preprocessing module: It is used for data input and standardization of the entire system, and is responsible for completing the entire process from raw facial video to standardized sequences that can be directly input into the model, providing high-quality, uniformly formatted data support for subsequent model training and validation.

[0051] The aforementioned facial video acquisition and data preprocessing module includes a video acquisition unit, a frame extraction unit, a face processing unit, and a standardization unit. The video acquisition unit can use a standard camera; all other units process data on a computer.

[0052] First, facial interview videos of the subjects are acquired through the video acquisition unit. The video format is standard MP4, with a resolution of no less than 640×480 and a frame rate of 30fps to ensure the clarity of facial features and the integrity of dynamic information. Then, the frame extraction unit decodes the video frame by frame using OpenCV tools, breaking the continuous video into a sequence of single-frame images. At the same time, the image sequence is initially screened to remove invalid frames, blurry frames, and black screen frames to ensure data quality.

[0053] The face processing unit employs the Dlib face detector and a 68-point facial landmark predictor to perform face region localization and pose alignment on a single-frame image sequence: see appendix. Figure 6 As shown, the face region in the image is first located by a face detector, and then the coordinates of key feature points such as facial contours and facial features are obtained by a facial key point predictor. Based on the key point coordinates, facial pose correction is completed to eliminate feature deviations caused by factors such as head rotation and tilt. For special scenarios where face detection fails, a fault-tolerant cropping mechanism is configured. The facial region is adaptively cropped by a preset facial center right offset and expansion cropping strategy to avoid data interruption due to detection failure.

[0054] The standardization unit uniformly crops and scales the aligned facial images to a size of 224×224 pixels, completing the standardization of spatial dimensions. At the same time, in order to preserve short-term facial dynamic details, a fixed-interval sampling method is used to extract video frames, and data augmentation operations such as random horizontal flipping, brightness adjustment, and contrast adjustment are introduced during the training phase to improve the model's generalization ability. Finally, a temporal segment containing 64 consecutive frames is generated as input data for model training. During the testing phase, no augmentation operations are added, and only the original frame sequence after cropping and alignment is output.

[0055] Please refer to Figure 3 , Figure 3 It is a diagram of a dual-branch, multi-scale 3D coordinate attention deep learning network structure, which includes: 3D convolutional layers and max pooling layers, masking layers, multi-scale 3D coordinate attention backbone branches, inter-frame difference dynamic feature extraction network, splicing layers, pooling layers and fully connected layers after splicing layers, and finally outputs a depression test score.

[0056] like Figure 4As shown, this network structure employs a multi-scale 3D coordinate attention backbone (M-CA). Traditional 3D convolutions weight all facial regions equally and can only fix the receptive field. The multi-scale 3D coordinate attention backbone can simultaneously model the height, width, and temporal 3D coordinates, adaptively enhancing key facial features. It achieves multi-scale feature extraction through dilated convolutions without significantly increasing the number of parameters. By simultaneously fusing spatial and temporal 3D positional information, it automatically focuses on facial feature regions and uses dilated convolutions to extract multi-scale static facial features at low cost, balancing accuracy and model lightweightness.

[0057] like Figure 5 As shown, the network structure designed includes an Inter-Frame Difference Dynamic Feature Extraction Network (TDDS). Single static convolutions struggle to capture subtle, low-speed micro-expressions on the faces of depressed individuals. This TDDS network explicitly calculates the pixel differences between frames, extracting depression-related micro-expression motion features through inter-frame difference operations. It specifically extracts short-term facial dynamic changes, with the output size strictly aligned with the main branches and the feature size unified with the main branches, achieving the fusion of complementary information from static facial structure and dynamic micro-expressions.

[0058] In this implementation example, a dual-branch parallel structure is constructed, using a standardized facial video sequence as input, to sequentially complete basic feature extraction, facial guided mask enhancement, dual-branch feature extraction, feature fusion, and depression score regression. First, basic spatiotemporal features are extracted through 3D convolutional layers, batch normalization layers, ReLU activation function and max pooling layer to output initial facial features, which are shallow features;

[0059] Subsequently, the facial guidance mask module is used to suppress background noise and enhance key facial features to generate dual-branch common input features. The main feature branch uses a 3D residual network as its backbone and embeds a multi-scale 3D coordinate attention module to extract facial static features with multiple receptive fields and spatiotemporal position awareness. The inter-frame difference dynamic branch extracts facial micro-expression dynamic features through temporal difference operations and scale calibration, and outputs dynamic features that match the size of the main branch. The dual-branch features are concatenated along the channel dimension, and then feature fusion is achieved through a 1×1×1 convolutional layer. Finally, the feature is mapped through adaptive pooling and a fully connected layer to output the depression score prediction result.

[0060] During the training and optimization of the dual-branch, multi-scale 3D coordinate attention deep learning network structure, a weighted fusion of mean squared error loss and smoothed L1 loss was used as the total loss function, with mean absolute error (MAE) and root mean square error (RMSE) as performance evaluation metrics. The optimizer employed the Adam algorithm with an initial learning rate of 0.001, dynamically adjusted via a cosine annealing restart mechanism to avoid oscillations in the later stages of training. A gradient accumulation strategy was also used, performing parameter updates every 10 steps of gradient accumulation to stabilize gradient noise in mini-batch training. To ensure experimental reproducibility, the global random seed was fixed at 42, the maximum number of training epochs was set to 200, and the batch size was set to 5. The performance on the validation set was monitored in real-time during training, and the model parameters for each epoch were saved. After training, the model with the lowest validation set error was selected as the optimal model for subsequent model validation and practical inference.

[0061] In practical engineering, the optimal model is loaded to predict the degree of depression in the facial video to be recognized, and the results are evaluated and interpreted to ensure the accuracy and reliability of the recognition results. Furthermore, the preprocessing unit of the module performs the same preprocessing operations as in the training phase on the video to be recognized: through frame extraction, face detection, key point alignment, and size normalization, the video is converted into a 224×224 pixel facial image sequence; for long video data, multiple consecutive 64-frame subsequences are uniformly extracted to ensure the integrity of temporal features.

[0062] The preprocessed subsequences are input segment by segment into the trained optimal model to obtain the depression score prediction result of a single subsequence; then the average of the prediction results of all subsequences is taken as the final depression score of the video.

[0063] Please refer to this again. Figure 2 , Figure 2 This is a structural diagram of a depression recognition method based on bi-branch multi-scale 3D coordinate attention according to the present invention. The steps in this embodiment include the following: This implementation example uses AVEC2013 and AVEC2014 as the benchmark datasets for the depression recognition task. Both datasets contain face interview video data labeled with BDI-II depression scores, providing standardized experimental support for model training, validation and evaluation.

[0064] The AVEC2013 dataset contains 150 individual video clips, officially divided into three parts: training set, validation set, and test set. The videos are stored in color MP4 format with a resolution of 640×480 and a frame rate of 30fps. The dataset uses the subjects' BDI-II depression scores as the ground truth, with a score range of 0 to 63. Based on the scores, the depression state is divided into four levels: mild, moderate, and severe, which can effectively distinguish different degrees of depression.

[0065] The AVEC2014 dataset contains 300 video samples, which are also divided into training, validation and test subsets. The data includes two collection task formats: fixed text reading task and free question and answer task. The subjects all completed the corresponding expressions in German. The scene is closer to the real interpersonal interview environment and can fully reflect the facial micro-expression and behavioral change characteristics in a natural state.

[0066] The experiments in this embodiment were conducted on a hardware platform equipped with an NVIDIA GeForce RTX 4060 graphics card, implemented using the PyTorch deep learning framework, and accelerated using CUDA. To ensure the reproducibility of the experimental results, the global random seed was fixed at 42. The batch size for model training was set to 5, and the maximum number of training epochs was 200. The optimizer used the Adam algorithm, with an initial learning rate of 0.001. The learning rate scheduling strategy adopted a cosine annealing restart mechanism. The mean absolute error and root mean square error were used as performance evaluation metrics for model training, and the calculation formulas were as follows: and .in For the sample size, To predict the score, This is a genuine BDI-II label.

[0067] Irrelevant background noise can severely interfere with depression feature extraction. Therefore, this embodiment designs a facial guidance mask module before feature bifurcation. This module aims to suppress activation of non-core regions and strengthen core regions highly correlated with depression by capturing the spatial saliency map of facial regions. Given features... The module first performs average pooling and max pooling along the channel dimension to extract global background and salient contour features in the spatial dimension:

[0068] To avoid excessive loss of original emotional features due to masking operations, the module employs a residual connection mechanism:

[0069] The multi-scale 3D coordinate attention module is the core module of this embodiment, aiming to solve the problems of single-scale and inaccurate positional localization in traditional feature extraction. For facial video depression recognition scenarios, this module accurately captures key facial features related to depression through multi-scale feature extraction and coordinate-aware design, strengthening the feature expression of key regions while avoiding the limitations of fixed receptive fields. This compensates for the shortcomings of traditional single-feature extraction methods, providing high-quality and targeted basic feature support for subsequent feature fusion and depression assessment. See Appendix. Figure 4 As shown, after the feature input is applied to this branch, it first undergoes the following transformation:

[0070] in The representation indicates that the expansion rate is i and the kernel size is... The 3D convolution operation is then performed. Subsequently, a dimension control layer is used to fuse and compress the concatenated features.

[0071] in, This indicates the concatenation of channel dimensions, and the kernel size is... 3D convolution operations. To compensate for the loss of positional information in global pooling, information is aggregated along three orthogonal axes: height, width, and time.

[0072]

[0073]

[0074] in, These represent the aggregated features in the height, width, and time dimensions, respectively. Next, the height feature is concatenated with the width feature after dimension permutation:

[0075] in This represents the concatenated features, and this represents the dimension permutation operation. Subsequently, the features are re-split along the spatial dimension into height and width features. Then, three independent convolutional layers are used to transform the aggregated features in the height, width, and time dimensions, respectively. The final output is:

[0076] in , , represents the weight vectors of the spatial vertical and horizontal coordinates, represents the dynamic weight vector of the time dimension, and represents the activation function.

[0077] Subsequently, after passing through the coordinate attention module, the multi-scale feature map is reweighted pixel-by-pixel using the obtained three orthogonal axis attention weights:

[0078] The output feature is deeply integrated with multi-scale receptive field and precise coordinate perception, realizing multi-scale dynamic modeling and precise coordinate localization of facial features related to depression.

[0079] To address the dynamic signs of facial expression evolution in patients with depression, this embodiment designs a temporal difference module. This branch aims to explicitly capture the motion residuals between adjacent frames at the feature level and extract high-dimensional motion representations through learnable downsampling operators, thereby complementing the static coordinate features of the main path.

[0080] See appendix Figure 5 As shown, given features As input, the time-domain difference operation is first performed to obtain the instantaneous motion vector:

[0081] in, This represents the captured facial motion changes. Subsequently, to reduce parameter overhead while preserving the diversity of dynamic features, 3D convolutional layers are used for feature pre-extraction:

[0082] Wherein, the kernel size is... The 3D convolution operation is then performed. This branch subsequently employs a two-stage concatenated stride convolution for learnable downsampling.

[0083] in and This represents a 3D convolutional downsampling module with a stride of (1,4,4).

[0084] To ensure dimensionality consistency during final feature fusion, an adaptive calibration mechanism is introduced at the end of the branch:

[0085] in, This represents the adaptive pooling operation, taking the feature output size from the 3D coordinate attention branch. This output compensates for the network's insufficient capture of dynamic features and achieves depth alignment between location-temporal features and temporal motion features through concatenation.

[0086] The multi-scale 3D coordinate attention branch generation feature is represented as follows: The branch generation feature of the difference module is represented as The two are combined, as shown in the following formula:

[0087] Finally, the features are input into the adaptive pooling layer and the fully connected layer to predict the depression score through a regression algorithm. The depression score ranges from 0 to 63. The higher the depression score, the greater the likelihood of depression.

[0088] Complete the depression identification and verification process: First, select the training model with the best MAE and RMSE indicators based on the validation set data, and use it as the optimal model for actual inference; then, perform the same frame extraction, face alignment, and standardized cropping preprocessing operations as in the training stage on the facial video to be identified.

[0089] Furthermore, during the verification process, multiple consecutive 64-frame subsequences were uniformly extracted from the preprocessed long video, and each was input into the optimal model to obtain a single-segment predicted score. Finally, the final depression score was calculated by averaging the results.

[0090] In conducting practical validation for depression identification, the facial video to be tested undergoes the same preprocessing operations as during the training phase. This includes frame extraction, face alignment, and size standardization to transform the video into a standardized sequence that meets the model's input requirements. Considering the integrity of the temporal features of long videos, the processed long video is evenly divided into multiple consecutive 64-frame subsequences. Each subsequence is input into the optimal model to predict the depression score. The average of the prediction results from all subsequences is then used as the final depression score for the video, ensuring the robustness and accuracy of the prediction results. Simultaneously, during validation, the mean absolute error and root mean square error (RMSE) are used to quantify the model's recognition accuracy, directly reflecting the difference between the predicted value and the actual BDI. II. The degree of bias in the depression score.

[0091] In addition, the Grad-CAM++ algorithm will be used to generate facial feature activation heatmaps, which will visualize the facial areas that the model focuses on during inference. This will allow for intuitive verification of whether the model accurately focuses on key facial areas that are highly related to the depressive state, such as the eyes and mouth, rather than irrelevant background or noisy areas. This will provide a double verification of the model's recognition reliability from the perspective of interpretability, ensuring that the recognition results are authentic, reliable, and based on evidence.

[0092] Therefore, existing models suffer from several drawbacks. Firstly, their modeling of facial features is relatively simplistic, making it difficult to accurately locate facial regions and key time segments highly correlated with depression. Secondly, their fixed receptive field convolutional structures cannot adapt to multi-scale dynamic changes in the face, resulting in insufficient attention to dynamic information and consequently, insufficient accuracy and interpretability in depression identification. This embodiment addresses these issues by using a residual network with an embedded multi-scale 3D coordinate attention module as the main branch. This network branch integrates multi-scale dilated convolution and 3D coordinate attention mechanisms, enabling it to capture multi-receptive field facial features and precise spatiotemporal location information. Simultaneously, a parallel inter-frame difference branch is constructed to explicitly extract dynamic facial expression features, and a facial guidance mask module is introduced to suppress background noise interference. Finally, the features from both branches are concatenated and fused, and end-to-end regression is used to accurately assess the degree of depression. This invention achieves relatively accurate depression identification, reduces identification errors, and ensures the model's interpretability and generalization, thus assisting in early screening for depression and population mental health monitoring.

[0093] Example 5 The purpose of this embodiment is to provide a computer program product containing instructions that, when run on a computer, causes the computer to perform the methods and functions involved in any of the embodiments described above.

[0094] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0095] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0096] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A depression recognition system based on bi-branch multi-scale 3D coordinate attention, characterized by: include: The facial video acquisition and data preprocessing module is configured to: acquire facial videos and perform face standardization through data preprocessing, and output face-standardized facial videos; The deep learning network module is configured to: construct a dual-branch, multi-scale 3D coordinate attention deep learning network, the network comprising: 3D convolutional layers and max pooling layers are used to extract basic spatiotemporal features from the input face-standardized facial video and output initial facial features; The masking layer is used to suppress background noise and enhance key facial features of the initial facial features, and to generate dual-branch common input features. The main feature branch uses a 3D residual network as its backbone and embeds a multi-scale 3D coordinate attention module to extract multi-receptive field, spatiotemporally aware facial static features from the mask layer output. The inter-frame difference dynamic branch extracts facial micro-expression dynamic features from the features output by the mask layer through temporal difference operations and scale calibration, and outputs dynamic features that match the size of the main feature branch. The concatenation layer concatenates the features of the two branches along the channel dimension, performs feature fusion through a convolutional layer, and then maps them through an adaptive pooling layer and a fully connected layer to output the depression prediction result.

2. The depression recognition system based on bi-branch multi-scale 3D coordinate attention as described in claim 1, characterized in that it further... include: The training optimization steps employ a weighted fusion of mean squared error loss and smoothed L1 loss as the total loss function, with mean absolute error and root mean square error used as performance evaluation metrics.

3. The depression recognition system based on bi-branch multi-scale 3D coordinate attention as described in claim 1, characterized in that, When suppressing background noise and enhancing key facial features in the initial facial features, the following steps are included: For a given feature, average pooling and max pooling are first performed along the channel dimension to extract global background and salient contour features in the spatial dimension; A residual connection mechanism is adopted: the given features are weighted and filtered using a mask, and then the weighted result is added back to the given features.

4. The depression recognition system based on bi-branch multi-scale 3D coordinate attention as described in claim 1, characterized in that, In the main feature branch, the input features are mapped. Then, the concatenated features are fused and compressed through the dimension control layer. After passing through the coordinate attention module, the multi-scale feature map is reweighted pixel by pixel using the obtained three orthogonal axis attention weights.

5. The depression recognition system based on bi-branch multi-scale 3D coordinate attention as described in claim 1, characterized in that, The process of standardizing faces through data preprocessing includes: Acquire facial interview videos to ensure the clarity of facial features and the completeness of dynamic information; The frame extraction unit decodes the video frame by frame, breaking the continuous video into a sequence of single-frame images. At the same time, the image sequence is initially screened to remove invalid frames, blurry frames, and black screen frames. The face processing unit uses a face detector and a facial key point predictor to perform face region localization and pose alignment on a single frame image. The aligned facial images are uniformly cropped and scaled to achieve spatial standardization; For scenarios where face detection fails, a fault-tolerant cropping mechanism is configured to adaptively crop the facial region through a preset strategy of rightward tilting of the face center and expanding the cropping area.

6. The depression recognition system based on bi-branch multi-scale 3D coordinate attention as described in claim 5, characterized in that, Face region localization and pose alignment for single-frame images: The face region in the image is located by a face detector, and the facial key point predictor is used to obtain the coordinates of the facial contour and key feature points of the facial features. Facial pose correction is completed based on the coordinates of the key feature points.

7. A depression identification method based on bi-branch multi-scale 3D coordinate attention, characterized by: include: Acquire facial videos and perform face standardization through data preprocessing, then output standardized facial videos; Construct a dual-branch, multi-scale 3D coordinate attention deep learning network, the network comprising: 3D convolutional layers and max pooling layers are used to extract basic spatiotemporal features from the input face-standardized facial video and output initial facial features; The masking layer is used to suppress background noise and enhance key facial features of the initial facial features, and to generate dual-branch common input features. The main feature branch uses a 3D residual network as its backbone and embeds a multi-scale 3D coordinate attention module to extract multi-receptive field, spatiotemporally aware facial static features from the mask layer output. The inter-frame difference dynamic branch extracts facial micro-expression dynamic features from the features output by the mask layer through temporal difference operations and scale calibration, and outputs dynamic features that match the size of the main feature branch. The concatenation layer concatenates the features of the two branches along the channel dimension, performs feature fusion through a convolutional layer, and then maps them through an adaptive pooling layer and a fully connected layer to output the depression prediction result.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of claim 6.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in claim 6.

Images