A depression detection method fusing space-time information, an electronic device and a storage medium

Abstract

The application provides a depression detection method fusing space-time information, an electronic device and a storage medium, and belongs to the computer medical cross field.The static global dynamic local space attention module, the bidirectional time attention module and the space-time fusion module are used;the static global dynamic local space attention module is combined with a self-attention mechanism and a convolution, captures a wide range of static space features while retaining local details, and solves the problem of dynamic information loss in single-frame image analysis;the bidirectional time attention module realizes collaborative perception of short-term micro-expression fluctuation and long-term emotional trend;and the space-time fusion module improves the effectiveness of space-time feature fusion through a channel and a space-time double dynamic screening mechanism.The application not only overcomes the subjective bias of manual feature extraction and the defects of a lack of a receptive field of a traditional deep model, but also realizes high-precision extraction of depression biomarkers hidden in a patient's facial expression, and provides a new method for automatic depression detection.

Description

Technical Field

[0001] This invention belongs to the field of computer medicine interdisciplinary technology, specifically relating to a depression detection method, electronic device, and storage medium that integrates spatiotemporal information. Background Technology

[0002] Major Depressive Disorder (MDD) is a common mood disorder. Its core symptoms include persistent pessimism, a significant loss of interest in daily activities or loss of pleasure, as well as other symptoms such as sleep disturbances, loss of appetite, and difficulty concentrating. Currently, the mainstream diagnostic method involves assessing the patient's level of depression through doctor interviews and depression scales. However, the number of qualified doctors is limited, especially in remote areas, where patients may have difficulty accessing timely medical consultation and diagnosis. Therefore, automated depression testing is receiving increasing attention.

[0003] Existing research has found that facial expression features of patients with depression differ from those of healthy individuals, thus visual cues have become important biomarkers for identifying depressive symptoms. In recent years, Automatic Depression Diagnosis (ADD) based on facial expressions has become an important method for identifying depression. Early ADD methods based on facial information mainly relied on manual feature extraction. To overcome the limitations of traditional manual feature extraction methods, deep learning techniques for analyzing the facial expressions of patients with depression have gradually become mainstream. Early deep learning-based depression diagnosis methods, such as DepressNet, attempted to identify depression using single-frame images. In real-world situations, a patient's expression may change with shifts in attention, making it difficult to accurately capture the patient's true emotional state using only a single frame. Therefore, identifying depression requires considering not only the complex spatial features of facial expressions but also capturing their dynamic changes. Another approach, such as Bi-LSTM, first extracts spatial features from videos and then performs ADD using temporal prediction models or multilayer perceptrons. Furthermore, because 3D convolutional neural networks can directly explore higher semantic associations in spatiotemporal information, they are currently widely used for early identification of depression.

[0004] Existing methods for assisting in the diagnosis of depression mainly fall into two categories: manual feature extraction methods and deep learning methods. While these methods have achieved some success in depression identification, they often rely on prior knowledge and self-organizing rules, potentially failing to fully capture deep semantic information within the data. Furthermore, their generalization ability is relatively limited, making them difficult to adapt to diverse data environments. Deep learning methods, on the other hand, rely heavily on the feature extraction techniques used. 3D convolutional neural networks, capable of directly exploring higher semantic connections in spatiotemporal information, are currently widely used for early depression identification. However, their inherent receptive field limitations make it difficult to capture a wide range of spatiotemporal dependencies. Moreover, the translation invariance of convolution prevents the model from highlighting important spatiotemporal features, resulting in insufficient ability to assess facial micro-expression changes in depressed patients, thus affecting the accuracy of depression diagnosis. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention proposes a depression detection method that integrates spatiotemporal information. This method aims to overcome the limitations of the receptive field and the insufficient spatiotemporal-dependent capture range of 3D convolutional neural networks. The method constructs an automatic depression diagnosis network, FST-Net, which fully utilizes the spatiotemporal information in video data to extract high-level information implicit in the data without disrupting the original spatiotemporal structure, thereby achieving accurate identification of depression.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0007] A method for detecting depression that integrates spatiotemporal information includes:

[0008] S1. Collect and preprocess the video and image data of the participants to generate a standardized dataset;

[0009] S2. Construct a Fusion of Spatio-Temporal Information Network for Depression Identification (FST-Net), which includes a Global Static and Local Dynamic Spatial Attention (GSLD) module, a Dual Temporal Attention (DTA) module, and a Spatial Temporal Fusion (STF) module.

[0010] The static global dynamic local spatial attention module processes the input image sequence. Global constant attention feature maps are obtained by using multi-head self-attention and convolutional normalization respectively. With local instantaneous features ,Will and Concatenation yields the feature vector The bidirectional temporal attention module runs in parallel with the static global dynamic local spatial attention module, processing the input image sequence. Parallel establishment of the interaction between the time and spatial dimensions yields time-highly correlated features. Time-width related features and will and Add to get the output The spatiotemporal fusion module processes the features obtained by the static global and dynamic local spatial attention modules. Features obtained with bidirectional temporal attention module To perform fusion, from feature maps and The system captures global temporal and spatial structure information respectively to obtain channel optimization features. With spatiotemporal weights Multiply them to obtain the final fused feature. ;

[0011] S3. The depression recognition network constructed in S2 is trained using the normalized dataset obtained in S1.

[0012] S4. Input the image of the patient with depression to be predicted into the network model trained in S3 to obtain the depression prediction result for the patient.

[0013] Furthermore, in S1, the specific process of generating the normalized dataset includes:

[0014] S1.1 Collect participants' facial features and retain video clips of them answering questions;

[0015] S1.2 For each participant's video data, the video is segmented into an image sequence consisting of several frames; and facial key points in each frame are extracted using Dlib, faces are aligned using affine transformations, and the images are scaled.

[0016] S1.3 Randomly vary the brightness, contrast, saturation, and hue of all frames in a video and randomly flip them horizontally to generate the final normalized dataset.

[0017] Furthermore, in the static global dynamic local spatial attention module (such as...) Figure 2 ), eigenvectors The calculation formula is:

[0018]

[0019] The The globally constant attention feature map obtained through multi-head attention mechanism is represented as follows:

[0020]

[0021] in, For multi-head attention operations; This indicates an average pooling operation;

[0022] The Extracted using a 1×3×3 convolutional layer The local instantaneous features are obtained by sliding a 3×3 convolution kernel along the time axis, followed by batch normalization, and finally, the ReLU activation function is selected to perform a nonlinear transformation on the output. This process is represented as:

[0023]

[0024] in, Represents the ReLU activation function; For batch normalization; This indicates that a 1×3×3 convolution extracts instantaneous features.

[0025] Furthermore, the global constant attention feature map The specific calculation process is as follows:

[0026] For the input image Average pooling is performed along the time dimension, and the time dimension is compressed to obtain static expression features. :

[0027]

[0028] in, Represents the number of channels. Indicates time depth. , These represent the height and width of each frame of the input video, respectively.

[0029] Subsequently, the multi-head attention mechanism MHSA was used to further extract static facial expression features. The formula for implementing the connection between different regions is as follows:

[0030]

[0031] in, It is the weight matrix used to synthesize all attention heads; It is the first The output of each attention head, It's the number of attention heads. It is the product of the image's width and height. The formula is:

[0032]

[0033] In the formula, Static facial expression features Different input projection weights obtained through mapping; This indicates that attention operations are performed on the features.

[0034] Furthermore, in the bidirectional temporal attention module (such as...) Figure 4 Time-highly correlated features The specific calculation process is as follows:

[0035] Compared to single static images, video data contains more complex and variable emotional information about patients; to better enhance temporal information and reduce information loss during temporal information extraction, the input image sequence... The bidirectional temporal attention module first focuses on... horizontal dimension Perform global average pooling and global max pooling operations separately, and then move the results along the horizontal dimension. By concatenating the vectors, we obtain the original horizontal aggregation vector. This process can be represented as:

[0036]

[0037] in, This indicates a global average pooling operation. This represents the global max pooling operation. Indicates along the horizontal dimension Perform the splicing operation;

[0038] Then aggregate the original horizontal vector The dimensions are rearranged as Then, a 3D convolution with a kernel size of 1 is used to further increase the horizontal dimension. The information is aggregated to obtain horizontal aggregated features. :

[0039]

[0040] in, This indicates a rearrangement of the vector dimensions. This represents a 3D convolutional layer with a kernel size of 1;

[0041] Subsequently, the bidirectional temporal attention module aggregates features horizontally through dimensionality compression. Dimensional compression to Then use a standard convolution pair with a kernel size of 3. Dimensions and The process of convolution on the dimension can be viewed as aggregating information from time-height features; then, after batch normalization and dimension expansion, the dimensions are rearranged. Attention weights are obtained through the Sigmoid activation function. :

[0042]

[0043] Finally, the original input and Multiplying them together yields time-highly correlated features. .

[0044] Furthermore, in the bidirectional temporal attention module, time-width correlation features The specific calculation process is as follows:

[0045] The bidirectional time attention module processes the input image sequence. vertical dimension Perform global average pooling and global max pooling operations separately, and then move the results along the vertical dimension. By concatenating the vectors, we obtain the original vertical aggregation vector. This process can be represented as:

[0046]

[0047] Then the original vertical aggregation vector The dimensions are rearranged as Then, a 3D convolution operation with a kernel size of 1 is used to obtain the vertical aggregation feature. :

[0048]

[0049] Subsequently, the bidirectional temporal attention module vertically aggregates features through dimensionality compression. Dimensional compression to Then use a standard convolution pair with a kernel size of 3. Dimensions and The dimensions are convolved, batch normalized, then expanded and rearranged to... Attention weights are obtained through the Sigmoid activation function. :

[0050]

[0051] Finally, the original input and Multiplying them together yields the time-width correlation feature. .

[0052] Furthermore, the spatiotemporal fusion module (such as...) Figure 5 Features obtained through the static global, dynamic local spatial attention module and the bidirectional temporal attention module and By fusing the data, channel optimization features are obtained. With spatiotemporal weights The two are then multiplied to obtain the final fused feature. The specific implementation process is as follows:

[0053] The spatiotemporal fusion module first combines the feature maps and The channels are concatenated; then global channel weights are obtained through global pooling, convolution operations, and the Sigmoid activation function. :

[0054]

[0055]

[0056] in, Indicates the dimension along the channel Perform the splicing operation. Indicates fusion characteristics, This indicates a global pooling operation;

[0057] Utilizing global channel weights Fusion features Calibration is performed; subsequently, a 1×1×1 convolution is used to select feature maps based on their importance; global channel weights are applied. Preserve the guiding convolutional layers By identifying key features and discarding less informative features, channel-optimized features are obtained. :

[0058]

[0059] To deeply model the spatiotemporal dependencies between local feature maps, the spatiotemporal fusion module uses standard convolution operations with a kernel size of 1×1×1 and the Sigmoid activation function to analyze the spatiotemporal interdependencies between feature maps. and The system captures global temporal and spatial structure information respectively, and their sum is the spatiotemporal weight. The spatiotemporal weights It is not only used to calibrate feature maps, but also to highlight regions that are significant in the spatiotemporal dimension, thereby improving the model’s performance in spatiotemporal feature fusion;

[0060]

[0061] Ultimately, the spatiotemporal weights will be... With channel optimization features Multiplying them together gives:

[0062]

[0063] in, This refers to the spatiotemporal fusion feature, i.e., the output of the spatiotemporal fusion module.

[0064] Furthermore, in S3, the SGD optimizer is used for training. A small batch of samples is randomly sampled from the training data, and the samples are processed through the depression recognition network to obtain fused features. The final sample prediction result is obtained after applying the activation function. and through Calculate the gradient of the loss function under the current network model parameters, and update the learning parameters according to the gradient direction and learning rate. Repeat the above training process, traversing the entire training dataset, until the loss function converges to a stable value.

[0065] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the electronic device performs the aforementioned method for detecting depression by incorporating spatiotemporal information.

[0066] A storage medium comprising a computer program that, when run on an electronic device, causes the electronic device to execute the aforementioned method for detecting depression by incorporating spatiotemporal information.

[0067] The beneficial effects of this invention: The depression recognition network FST-Net proposed in this invention overcomes the limitations of traditional 3D convolutional neural networks in capturing spatiotemporal features by integrating Global Static and Local Dynamic Spatial Attention (GSLD), Bidirectional Temporal Attention (DTA), and Spatiotemporal Dynamic Selection (STF) mechanisms. The GSLD module combines self-attention with convolution to capture large-scale static spatial features while preserving local details, solving the problem of dynamic information loss in single-frame image analysis. The DTA module integrates a bidirectional attention mechanism with positional encoding to achieve collaborative perception of short-term micro-expression fluctuations and long-term emotional trends. The STF module significantly improves the effectiveness of spatiotemporal feature fusion through a dual dynamic selection mechanism of channels and spatiotemporal features. The superiority of this invention is verified on a constructed depression video dataset. It not only overcomes the subjective bias of manual feature extraction and the shortcomings of insufficient receptive field in traditional deep models, but also achieves high-precision extraction of depression biomarkers implicit in patients' facial expressions through hierarchical dynamic selection of spatiotemporal features, providing a new method with both robustness and generalization ability for automated depression detection. Attached Figure Description

[0068] Figure 1 This is an overall architecture diagram of the FST-Net automatic depression diagnosis network in an embodiment of the present invention.

[0069] Figure 2 This is a structural diagram of the GSLD module in an embodiment of the present invention.

[0070] Figure 3 This is a diagram illustrating common methods for processing time information related to depression.

[0071] Figure 4 This is a structural diagram of the DTA module in an embodiment of the present invention.

[0072] Figure 5 This is a structural diagram of the STF module in an embodiment of the present invention.

[0073] Figure 6 Examples of visualizations of heatmaps at different times are provided; (a) is the attention heatmap of non-depressed samples correctly predicted by the model at different time segments, (b) is the attention heatmap of depressed samples correctly predicted by the model wearing masks at different time segments, and (c) is the attention heatmap of depressed samples correctly predicted by the model not wearing masks at different time segments. Detailed Implementation

[0074] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0075] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for detecting depression by fusing spatiotemporal information, including the following steps:

[0076] Step 1: Collect video and image data from participants and preprocess it to generate a standardized dataset.

[0077] Data collection for this embodiment was conducted in the soundproof outpatient clinic of the Department of Psychiatry and Psychology, Affiliated Hospital of Guangdong Medical University. All subjects in the case group met the criteria of the Diagnostic and Statistical Manual of Mental Disorders (DSM). The control group consisted of healthy individuals with no history of mental illness. The 24-item Hamilton Depression Rating Scale (HAMD) was used to determine whether participants met the inclusion and experimental criteria. The HAMD scale can be used for the auxiliary diagnosis of depression and to assess the severity of depressive symptoms. During the experiment, two physicians simultaneously guided the subjects. This embodiment was approved by the Ethics Review Committee of the China Registered Clinical Trials and was conducted strictly in accordance with the ethical guidelines of the World Medical Association (Declaration of Helsinki). Before data collection, all subjects understood the experimental details and confirmed their consent to participate.

[0078] The experiment included 365 participants, comprising 173 patients with depression and 192 without depression. Video was captured using high-resolution cameras. The cameras were placed on the walls to the left and right of the examining physician, with the camera angle adjusted for each participant to ensure optimal recording of facial features. The video frame rate was set to 30fps. The physician's speaking segments were removed from the full video, retaining only the video clips of each patient answering questions.

[0079] The dataset was divided into a training set, a validation set, and a test set using random stratified sampling, containing 217, 74, and 74 samples respectively. The ratio of positive to negative samples in each set is shown in Table 1.

[0080] Table 1. Distribution of positive and negative samples in the dataset

[0081]

[0082] To eliminate the influence of useless background information on the experimental results, for each sample of video data in the dataset, facial keypoints were extracted from each frame using Dlib, faces were aligned using affine transformations, and the image size was scaled to 224×224. To improve the generalization and robustness of the model, the brightness, contrast, saturation, and hue of all frames in a video clip were randomly varied, and the frames were randomly flipped horizontally; during the testing phase, cropped and aligned video frames were used without any enhancements. For a given test video, it was cropped into 64-frame sub-videos, and the average probability of depression was calculated for each segment, thus producing the final result for that video.

[0083] Step 2: Construct and train FST-Net, a depression recognition network that integrates spatiotemporal information.

[0084] A depression recognition network, FST-Net, was designed and implemented using the PyTorch deep learning framework. The network model was trained using the SGD optimizer on an NVIDIA Tesla V100 GPU with 16GB of VRAM. The training iterations were set to 200 epochs, with a batch size of 8, a learning rate of 0.0001, a momentum of 0.9, and a weight decay of 0.0001. To improve the model's generalization ability and prevent overfitting, the network weights were saved when the validation loss reached its historical minimum.

[0085] Choosing the commonly used cross-entropy loss function for classification tasks, for binary classification tasks, the cross-entropy loss can be written as:

[0086]

[0087] in, For true labels (0: non-depressed, or 1: depressed), It is the probability that the model predicts the patient has depression.

[0088] Table 2 Confusion Matrix for Depression Classification

[0089]

[0090] Precision, recall, and F1 score were selected as the performance metrics for the model. To calculate these metrics, the depression classification confusion matrix shown in Table 2 was constructed.

[0091] Precision measures the proportion of actual positive cases that the model predicts as positive. It reflects the accuracy of the model's predictions, and the calculation formula is as follows:

[0092]

[0093] Recall measures the proportion of samples that are actually positive, and the model correctly predicts them as positive. The formula is as follows:

[0094]

[0095] The F1 score is the harmonic mean of precision and recall. It strikes a balance between the two, providing a comprehensive performance metric that considers both precision and recall. The calculation formula is as follows:

[0096]

[0097] To verify the effectiveness of FST-Net, it was compared with state-of-the-art methods in the research field. Since there are relatively few open-source video-based depression detection methods, this invention also selected some widely recognized state-of-the-art methods in video understanding tasks as comparison models. The comparison methods mainly fall into two categories: video recognition-based depression detection methods and video understanding methods. Video recognition-based depression detection methods mainly include STA, MSN, and MDN; video understanding methods include the widely recognized classic model SlowFast, as well as advanced methods VideoMAE2 and vid-TLDR. For these models, if pre-trained weights are provided, the pre-trained model will be used and fine-tuned. For SlowFast, VideoMAE2, and vid-TLDR, the number of input frames is set to 16; otherwise, the pre-trained weights cannot be utilized. The above training process settings were used in the comparison and ablation experiments in Tables 3 and 4.

[0098] Table 3 shows a comparison with state-of-the-art methods on benchmark datasets.

[0099]

[0100] Table 3 shows the comparative experimental results of FST-Net and the contrasting methods on the benchmark dataset, where bold values ​​represent the best values ​​and underlined values ​​represent the second-best values. The results clearly show that FST-Net significantly outperforms other contrasting methods in the depression recognition task. Specifically, FST-Net achieves precision, recall, and F1 score of 91.67%, 94.29%, and 92.96%, respectively, obtaining two of the best values ​​across the three evaluation metrics. Although FST-Net's F1 score is slightly lower than vid-TLDR, vid-TLDR's recall is only 82.86%. Overall, FST-Net demonstrates a comprehensive performance advantage compared to other methods. Transformer-based video understanding models such as VideoMAE2 and vid-TLDR only achieve F1 scores of 88.89% and 87.88%, respectively. This may be due to their lack of some inductive biases inherent in CNNs, namely, their perception of locality, requiring large datasets to achieve good results. The STA model, which also employs a temporal attention mechanism, performs dimensionality reduction on the data in both the vertical (height) and horizontal (width) directions when processing temporal information. The FST-Net model, however, takes a more comprehensive approach, utilizing two separate branches to handle the relationship between height and time, and width and time, respectively. This design allows FST-Net to capture and analyze temporal features in videos with greater detail, resulting in a 3.1% higher F1 score than the STA model. Compared to the MDN model, FST-Net provides a larger receptive field, compensating for the weakness of 3D CNNs in capturing long-range spatial dependencies, and improving the F1 score by 2.82%.

[0101] Secondly, regarding model size, compared to vid-TLDR (64.97M) with the largest parameter size and MSN (696.65 GFlops) with the largest computational cost, FST-Net achieves significant performance improvements with fewer parameters and lower computational cost. Compared to vid-TLDR, it improves recall by 11.43% and F1 score by 5.08%; compared to the MSN model, it leads in all three metrics, with improvements in precision by 5.56%, recall by 5.72%, and F1 score by 5.64%. Furthermore, although FST-Net has significantly higher computational complexity than SlowFast (R50) with its 1.482M parameters, this increased computational requirement significantly improves the model's predictive performance, allowing FST-Net to outperform SlowFast (R50) in all three evaluation metrics. Therefore, the advantage of this invention lies in its ability to provide performance comparable to or even better than pre-trained models while maintaining lower computational costs. This means that compared to comparative methods, FST-Net is more in line with the needs of real-world applications and is more suitable for clinical use.

[0102] To comprehensively evaluate the performance of the GSLD and DTA modules under different combinations, a series of comparative experiments were conducted, involving six different application methods. The experimental results are shown in Table 4.

[0103] When used independently, the GSLD and DTA modules achieved F1 scores of 85.30% and 87.32%, respectively. Combining them in a sequential cascade manner improved the F1 scores to varying degrees. Firstly, using the GSLD module followed by the DTA module improved the F1 scores by 3.59% and 1.57% respectively compared to using either module alone. Similarly, using the DTA module first followed by the GSLD module improved the F1 scores by 3.27% and 1.25% respectively. This demonstrates that these two modules significantly improve the model's predictive performance by focusing on key spatial and temporal features in the video data. Specifically, the GSLD module, by combining static global features and dynamic local features, can capture subtle changes in the facial expressions of patients with depression, changes that are often imperceptible to human observers. The DTA module, on the other hand, reveals the temporal dynamics of behavioral patterns in patients with depression by analyzing pixel changes over time in the vertical and horizontal directions.

[0104] Table 4 compares the performance of different module combinations. "+" indicates that modules are used in a sequential cascade manner. This indicates that modules are used in parallel.

[0105]

[0106] Furthermore, using the GSLD and DTA modules in parallel yields better performance, improving the F1 score by 2.54% and 2.86% respectively compared to sequentially combining the two modules. When combining modules sequentially, the model may lose information during feature propagation. The parallel structure reduces this information loss because each module can independently extract and utilize temporal or spatial information.

[0107] Further experiments show that using a better fusion strategy can further improve the model's performance. The model's prediction performance is optimal when the outputs of the GSLD and DTA modules are fused through the STF module, achieving an F1 score of 92.96%, a 1.53% improvement over simply adding spatial and temporal features. The STF module dynamically fuses these features, enabling the model to more comprehensively understand the spatiotemporal information in the video data. This spatiotemporal fusion method improves performance more than using temporal or spatial features alone because it simultaneously captures the static features of facial expressions and the temporal dynamics of behavior in patients with depression. Furthermore, compared to simple additive fusion, the STF module employs a more complex fusion strategy, allowing the model to more effectively utilize the spatiotemporal features in the data, thus achieving more accurate depression predictions.

[0108] Figure 6 This demonstrates the results of visualizing consecutive frames of different samples using the general heatmap visualization technique Grad-CAM. Grad-CAM generates a heatmap by analyzing the network's gradients, thereby highlighting key regions in the image. By overlaying this heatmap onto the original image, the areas of interest to the model can be visually identified. This embodiment modifies Grad-CAM to adapt it to the needs of video data analysis.

[0109] exist Figure 6 (a) shows a sample that was determined not to have depression; the heatmap shows low activation intensity, indicating that the model did not detect significant features associated with depression in these areas. In contrast, Figure 6 (b) and (c) show samples correctly identified as having depression; in these samples, the heatmaps showed higher activation intensity, particularly in the areas around the eyes, nose, and mouth. Notably, even in Figure 6 (b) Even with subjects wearing masks, FST-Net was still able to accurately identify key facial features. The heatmap showed high activation intensity in the area around the eyes, indicating that the model was able to extract effective features from limited facial information. Figure 6 (c) shows a sample of depression without occlusion. The heatmap covers a wider facial area, including the forehead, nose and mouth, demonstrating that the model can comprehensively analyze facial expressions and capture the correlation between different parts.

[0110] Analysis of these visualizations leads to the following conclusions: First, FST-Net demonstrates excellent feature extraction capabilities. The model can accurately locate and analyze key facial regions, particularly the eyes, nose, and mouth, which play a crucial role in the manifestations of depression. This precise feature localization provides a solid foundation for accurately identifying depressive symptoms. Second, FST-Net exhibits strong adaptability and robustness. Even with partial facial occlusion, such as... Figure 6 As shown in image (b), even with a mask on, the model is still able to extract sufficient information from the visible facial area to make an accurate judgment. Finally, FST-Net is able to capture dynamic features. By analyzing consecutive video frames, the model not only focuses on static facial features but also captures subtle changes in facial expressions over time. This dynamic analysis capability allows the model to more comprehensively assess the patient's emotional state, providing richer and more accurate information than single static image analysis.

[0111] Finally, it should be noted that the above embodiments are intended to illustrate the technical solutions of the present invention and do not constitute any limitation on the present invention. Those skilled in the art should fully understand that modifications to the technical solutions described in the foregoing embodiments or equivalent substitutions for any part or all of the technical features are entirely feasible. Such modifications or substitutions, as long as they do not depart from the scope of protection defined by the claims of the present invention, should be considered reasonable extensions of the present invention.

Claims

1. A method for detecting depression that integrates spatiotemporal information, characterized in that, include: S1. Collect and preprocess the video and image data of the participants to generate a standardized dataset; S2. Construct a depression recognition network that integrates spatiotemporal information. This network includes a static global and dynamic local spatial attention module, a bidirectional temporal attention module, and a spatiotemporal fusion module. The static global dynamic local spatial attention module processes the input image sequence. Global constant attention feature maps are obtained by using multi-head self-attention and convolutional normalization respectively. With local instantaneous features ,Will and Concatenation yields the feature vector The bidirectional temporal attention module runs in parallel with the static global dynamic local spatial attention module, processing the input image sequence. Parallel establishment of the interaction between the time and spatial dimensions yields time-highly correlated features. Time-width related features and will and Adding them together yields the output feature vector. The spatiotemporal fusion module processes the feature vectors obtained by the static global and dynamic local spatial attention modules. Feature vectors obtained with bidirectional temporal attention module To perform fusion, from feature vectors and The system captures global temporal and spatial structure information respectively to obtain channel optimization features. With spatiotemporal weights Multiply them to obtain the final fused feature. ; S3. The depression recognition network constructed in S2 is trained using the normalized dataset obtained in S1. S4. Input the image of the patient with depression to be predicted into the network model trained in S3 to obtain the depression prediction result of the patient. In the static global dynamic local spatial attention module, the feature vector The calculation formula is: The The globally constant attention feature map obtained through multi-head attention mechanism is represented as follows: in, For multi-head attention operations; This represents the average pooling operation; the global constant attention feature map The specific calculation process is as follows: For the input image sequence Average pooling is performed along the time dimension, and the time dimension is compressed to obtain static expression features. : in, Represents the channel dimension. Indicates time depth. , These represent the vertical and horizontal dimensions of the input image sequence, respectively. Subsequently, the multi-head attention mechanism MHSA was used to further extract static facial expression features. The connection between different regions in the process is as follows: in, It is the weight matrix used to synthesize all attention heads; It is the first The output of each attention head, It's the number of attention heads. It is the product of the image's width and height. The formula is: In the formula, Static facial expression features Different input projection weights obtained through mapping; This indicates that attention operations are performed on the features. The Extracted using a 1×3×3 convolutional layer The local instantaneous features are then extracted, batch normalization is performed, and finally, the ReLU activation function is selected to perform a nonlinear transformation on the output. This process is expressed as follows: in, Represents the ReLU activation function; For batch normalization; This indicates that a 1×3×3 convolution extracts instantaneous features; In the bidirectional temporal attention module, time-highly correlated features The specific calculation process is as follows: For the input image sequence The bidirectional temporal attention module first focuses on... horizontal dimension Perform global average pooling and global max pooling operations separately, and then move the results along the horizontal dimension. By concatenating the vectors, we obtain the original horizontal aggregation vector. The process is represented as: in, This indicates a global average pooling operation. This represents the global max pooling operation. Indicates along the horizontal dimension Perform the splicing operation; Then aggregate the original horizontal vector The dimensions are rearranged as Then, a 3D convolution with a kernel size of 1 is used to further increase the horizontal dimension. The information is aggregated to obtain horizontal aggregated features. : in, This indicates a rearrangement of the vector dimensions. This represents a 3D convolutional layer with a kernel size of 1; Subsequently, the bidirectional temporal attention module aggregates features horizontally through dimensionality compression. Dimensional compression to Then use a standard convolution pair with a kernel size of 3. Dimensions and The process involves convolution of the dimensions, which aggregates information from time-height features; then, after batch normalization and dimensional expansion, the dimensions are rearranged. Attention weights are obtained through the Sigmoid activation function. : Finally, the image sequence and Multiplying them together yields time-highly correlated features. ; In the bidirectional temporal attention module, time-width correlation features The specific calculation process is as follows: The bidirectional temporal attention module applies to the input image sequence vertical dimension Perform global average pooling and global max pooling operations separately, and then move the results along the vertical dimension. By concatenating the vectors, we obtain the original vertical aggregation vector. The process is represented as: Then the original vertical aggregation vector The dimensions are rearranged as Then, a 3D convolution operation with a kernel size of 1 is used to obtain the vertical aggregation feature. : Subsequently, the bidirectional temporal attention module vertically aggregates features through dimensionality compression. Dimensional compression to Then use a standard convolution pair with a kernel size of 3. Dimensions and The dimensions are convolved, batch normalized, then expanded and rearranged to... Attention weights are obtained through the Sigmoid activation function. : Finally, the image sequence and Multiplying them together yields the time-width correlation feature. ; In the spatiotemporal fusion module, the fusion features The specific implementation process is as follows: The spatiotemporal fusion module first combines the feature vectors and The channels are concatenated; then global channel weights are obtained through global pooling, convolution operations, and the Sigmoid activation function. : in, Indicates the dimension along the channel Perform the splicing operation. Indicates fusion characteristics, This indicates a global pooling operation; Utilizing global channel weights Fusion features Perform calibration, global channel weights Preserve the guiding convolutional layers Key features, yielding channel optimization features : The spatiotemporal fusion module uses standard convolution operations with a kernel size of 1×1×1 and the Sigmoid activation function to extract features from the feature vector. and The system captures global temporal and spatial structure information respectively, and their sum is the spatiotemporal weight. : Ultimately, the spatiotemporal weights will be... With channel optimization features Multiplication: in, It is a spatiotemporal fusion feature.

2. The method for detecting depression by fusing spatiotemporal information according to claim 1, characterized in that, In S1, the specific process of generating the normalized dataset includes: S1.1 Collect participants' facial features and retain video clips of them answering questions; S1.2 For each participant's video data, the video is divided into an image sequence consisting of several frames; and facial key points in each frame are extracted using Dlib, faces are aligned using affine transformation, and the images are scaled. S1.3 Randomly vary the brightness, contrast, saturation, and hue of all frames in a video and randomly flip them horizontally to generate the final normalized dataset.

3. The method for detecting depression by fusing spatiotemporal information according to claim 1, characterized in that, In step S3, the SGD optimizer is used for training. A small batch of samples is randomly sampled from the training data, and the samples are processed through a depression recognition network to obtain fused features. The final sample prediction result is obtained after applying the activation function. and through Calculate the gradient of the loss function under the current network model parameters, and update the learning parameters according to the gradient direction and learning rate. Repeat the above training process, traversing the entire training dataset, until the loss function converges to a stable value.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor; characterized in that, When the processor executes the computer program, the electronic device performs a depression detection method that integrates spatiotemporal information as described in any one of claims 1 to 3.

5. A storage medium comprising a computer program that, when run on an electronic device, causes the electronic device to perform a depression detection method incorporating spatiotemporal information as described in any one of claims 1 to 3.

Images