Spatial-temporal information fused depression detection method, electronic equipment and storage medium

Abstract

The invention provides a depression detection method fusing spatio-temporal information, electronic equipment and a storage medium, and belongs to the field of computer medical crossing. The system comprises a static global dynamic local space attention module, a bidirectional time attention module and a space-time fusion module; the static global dynamic local space attention module combines a self-attention mechanism with convolution, captures large-range static space features while retaining local details, and solves the problem of dynamic information loss in single-frame image analysis; the bidirectional time attention module realizes collaborative perception of short-term micro-expression fluctuation and long-term emotion trend; the space-time fusion module improves the effectiveness of space-time feature fusion through a channel and space-time dual dynamic screening mechanism. The method not only overcomes the defects of subjective deviation of manual feature extraction and insufficient receptive field of a traditional depth model, but also realizes high-precision extraction of depression biomarkers implied in facial expressions of a patient, and provides a new method for automatic depression detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer medicine intersection, and particularly relates to a depression detection method fusing space-time information, an electronic device and a storage medium. BACKGROUND

[0002] Depression (Major Depressive Disorder, MDD) is a common mood disorder. Its core symptoms include persistent pessimistic mood, significant decrease in interest in daily activities or loss of pleasure, and other symptoms such as sleep disorders, loss of appetite, difficulty concentrating, etc. The current mainstream diagnosis method is to judge the depression level of patients through doctor interviews and depression scales. However, the number of professional doctors is limited, especially in some remote areas, patients may have difficulty in obtaining timely medical consultation and diagnosis. Therefore, automatic depression detection has gradually attracted more and more attention.

[0003] Previous studies have found that the facial expression features of patients with depression are different from those of healthy individuals, so visual cues have become an important biomarker for identifying depression symptoms. In recent years, automatic depression diagnosis (ADD) through facial expressions has become an important method for identifying depression. Early ADD methods based on facial information mainly rely on manual feature extraction. In order to break through the limitations of traditional manual feature extraction methods, deep learning technology has gradually become the mainstream method for analyzing the facial expressions of patients with depression. Among them, early depression diagnosis methods based on deep learning such as DepressNet attempt to use single-frame image-based methods to identify depression. In actual situations, the patient's expression may change with the shift of attention, and it is difficult to accurately capture the patient's true emotional state with only single-frame images. Therefore, when identifying depression, not only the spatial features of complex facial expressions should be considered, but also the dynamic changes of facial expressions. Another method such as Bi-LSTM first extracts spatial features from video, and then uses time series prediction model or multilayer perception method to perform ADD. In addition, since 3D convolutional neural network can directly explore higher semantic associations in space-time information, it has been widely used in early depression recognition.

[0004] Existing methods for assisting in the diagnosis of depression mainly fall into two categories: manual feature extraction methods and deep learning methods. While these methods have achieved some success in depression identification, they often rely on prior knowledge and self-organizing rules, potentially failing to fully capture deep semantic information within the data. Furthermore, their generalization ability is relatively limited, making them difficult to adapt to diverse data environments. Deep learning methods, on the other hand, rely heavily on the feature extraction techniques used. 3D convolutional neural networks, capable of directly exploring higher semantic connections in spatiotemporal information, are currently widely used for early depression identification. However, their inherent receptive field limitations make it difficult to capture a wide range of spatiotemporal dependencies. Moreover, the translation invariance of convolution prevents the model from highlighting important spatiotemporal features, resulting in insufficient ability to assess facial micro-expression changes in depressed patients, thus affecting the accuracy of depression diagnosis. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention proposes a depression detection method that integrates spatiotemporal information. This method aims to overcome the limitations of the receptive field and the insufficient spatiotemporal-dependent capture range of 3D convolutional neural networks. The method constructs an automatic depression diagnosis network, FST-Net, which fully utilizes the spatiotemporal information in video data to extract high-level information implicit in the data without disrupting the original spatiotemporal structure, thereby achieving accurate identification of depression.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0007] A method for detecting depression that integrates spatiotemporal information includes:

[0008] S1. Collect and preprocess the video and image data of the participants to generate a standardized dataset;

[0009] S2. Construct a Fusion of Spatio-Temporal Information Network for Depression Identification (FST-Net), which includes a Global Static and Local Dynamic Spatial Attention (GSLD) module, a Dual Temporal Attention (DTA) module, and a Spatial Temporal Fusion (STF) module.

[0010] The static global dynamic local spatial attention module processes the input image sequence. The global constant attention feature map is obtained by using multi-head self-attention and convolution normalization respectively and local instantaneous features , and are spliced to obtain a feature vector ; the bidirectional time attention module and the static global dynamic local spatial attention module are parallel, and the image sequence input is used to establish the interaction between the time dimension and the space dimension in parallel, to obtain time-height correlation features and time-width correlation features , and the output is obtained by adding ; The spatio-temporal fusion module fuses the features obtained by the static global dynamic local spatial attention module and the features obtained by the bidirectional time attention module , to capture global time and space structure information from the feature maps and , to obtain channel-optimized features and spatio-temporal weights , and the final fusion features are obtained by multiplying them;

[0011] S3, using the normalized data set obtained in S1 to train the depression recognition network constructed in S2;

[0012] S4, input the image of the patient to be predicted into the network model trained in S3 to obtain the depression prediction result of the patient.

[0013] Further, in the S1, the specific process of generating the normalized data set includes:

[0014] S1.1, collect the facial features of the participants and retain the video clips when answering the questions;

[0015] S1.2, for each participant's video data, the video is cut into an image sequence composed of a plurality of frames of images; and the facial key points in each frame of image are extracted using Dlib, the face is aligned using affine transformation, and the image is scaled;

[0016] S1.3, randomly change the brightness, contrast, saturation and hue of all frames in a video, and randomly perform horizontal flip to generate the final normalized data set.

[0017] Further, in the static global dynamic local spatial attention module (such as Figure 2 ), the calculation formula of the feature vector is:

[0018]

[0019] The The global constant attention feature map calculated by the multi-head attention mechanism is represented as:

[0020]

[0021] wherein, is a multi-head attention operation; represents an average pooling operation;

[0022] The is a local instantaneous feature of the extracted using a 1x3x3 convolutional layer, that is, a 3x3 convolution kernel is used for sliding on the time axis, then batch normalization is performed, and finally a ReLU activation function is selected to perform nonlinear transformation on the output, and the process is represented as:

[0023]

[0024] wherein, represents a ReLU activation function; is batch normalization; represents a 1x3x3 convolutional instantaneous feature extraction.

[0025] Further, the global constant attention feature map is calculated as follows:

[0026] The input image is subjected to an average pooling operation along the time dimension and compressed along the time dimension to obtain a static expression feature :

[0027]

[0028] wherein, represents the number of channels, represents the time depth, , respectively represent the height and width of each frame of the input video image;

[0029] Subsequently, a multi-head attention mechanism MHSA is used to further extract the correlation of different regions in the static expression feature , and the implementation formula of the process is:

[0030]

[0031] wherein, is a weight matrix for synthesizing all attention heads; is the i-th The output of each attention head, It's the number of attention heads. It is the product of the image's width and height. The formula is:

[0032]

[0033] In the formula, Static facial expression features Different input projection weights obtained through mapping; This indicates that attention operations are performed on the features.

[0034] Furthermore, in the bidirectional temporal attention module (such as...) Figure 4 Time-highly correlated features The specific calculation process is as follows:

[0035] Compared to single static images, video data contains more complex and variable emotional information about patients; to better enhance temporal information and reduce information loss during temporal information extraction, the input image sequence... The bidirectional temporal attention module first focuses on... horizontal dimension Perform global average pooling and global max pooling operations separately, and then move the results along the horizontal dimension. By concatenating the vectors, we obtain the original horizontal aggregation vector. This process can be represented as:

[0036]

[0037] in, This indicates a global average pooling operation. This represents the global max pooling operation. Indicates along the horizontal dimension Perform the splicing operation;

[0038] Then aggregate the original horizontal vector The dimensions are rearranged as Then, a 3D convolution with a kernel size of 1 is used to further increase the horizontal dimension. The information is aggregated to obtain horizontal aggregated features. :

[0039]

[0040] in, This indicates a rearrangement of the vector dimensions. This represents a 3D convolutional layer with a kernel size of 1;

[0041] Then, the bidirectional time attention module compresses the dimension of the horizontal aggregated feature to , and then uses a standard convolution with a kernel size of 3 to convolve the dimension of with the dimension of , which can be regarded as information aggregation of the time-high feature; then, batch normalization and dimension expansion are performed, and the dimension is rearranged to , and the attention weight is obtained through the Sigmoid activation function:

[0042]

[0043] Finally, the original input is multiplied by to obtain the time-high correlation feature .

[0044] Further, the specific calculation process of the time-width correlation feature in the bidirectional time attention module is as follows:

[0045] The bidirectional time attention module performs global average pooling and global maximum pooling operations on the vertical dimension of the input image sequence respectively, then splices the results along the vertical dimension to obtain the original vertical aggregated vector , which can be represented as:

[0046]

[0047] Then, the dimension of the original vertical aggregated vector is rearranged to , and a 3D convolution operation with a kernel size of 1 is performed to obtain the vertical aggregated feature :

[0048]

[0049] Then, the bidirectional time attention module compresses the dimension of the vertical aggregated feature to , and then uses a standard convolution with a kernel size of 3 to convolve the dimension of with the dimension of , and performs batch normalization, then expands and rearranges the dimension to , and the attention weight is obtained through the Sigmoid activation function:

[0050]

[0051] Finally, the original input is multiplied by the time-width correlation feature to obtain the final feature .

[0052] Further, the spatio-temporal fusion module (such as Figure 5 ) fuses the features obtained by the static global dynamic local spatial attention module and the bidirectional time attention module to obtain the channel-optimized feature and the spatio-temporal weight , and multiplies the two to obtain the final fusion feature . The specific implementation process is as follows:

[0053] The spatio-temporal fusion module first splices the feature maps and along the channel; then, through global pooling, convolution operation and Sigmoid activation function processing, the global channel weight is obtained:

[0054]

[0055]

[0056] wherein represents the splicing operation along the channel dimension , represents the fusion feature, represents the global pooling operation. The global channel weight

[0057] is used to calibrate the fusion feature ; subsequently, a 1×1×1 convolution is used to select the feature map according to the importance of the feature map; the global channel weight will guide the convolution layer to retain the important features and discard the features with less information, thereby obtaining the channel-optimized feature :

[0058] In order to deeply model the spatio-temporal interdependence between local feature maps, the spatio-temporal fusion module captures the global time and space structure information from the feature maps and

[0059] through a standard convolution operation with a kernel size of 1×1×1 and a Sigmoid activation function, and their sum is the spatio-temporal weight , which is used to guide the convolution layer to retain the important features and discard the features with less information, thereby obtaining the channel-optimized feature : ​Not only used for calibration feature map, but also emphasized the region with significant in spatio-temporal dimension, thus improved the performance of model in spatio-temporal feature fusion;

[0060]

[0061] Finally, the spatio-temporal weight is multiplied by the channel optimization feature to obtain:

[0062]

[0063] wherein, is the spatio-temporal fusion feature, i.e. the output of the spatio-temporal fusion module.

[0064] Further, in the S3, the SGD optimizer is used for training, a small batch sample is randomly sampled from the training data, the fusion feature obtained after the sample passes through the depression recognition network is obtained by the activation function to obtain the final sample prediction result , and the loss function gradient under the current network model parameters is calculated , and the learning parameters are updated according to the gradient direction and the learning rate, the above training process is repeated, and the entire training data set is traversed until the loss function converges to a stable value.

[0065] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor; when the processor executes the computer program, the electronic device executes the depression detection method of fusing spatio-temporal information.

[0066] A storage medium comprising a computer program, when the computer program is executed on an electronic device, the electronic device executes the depression detection method of fusing spatio-temporal information.

[0067] The beneficial effects of this invention: The depression recognition network FST-Net proposed in this invention overcomes the limitations of traditional 3D convolutional neural networks in capturing spatiotemporal features by integrating Global Static and Local Dynamic Spatial Attention (GSLD), Bidirectional Temporal Attention (DTA), and Spatiotemporal Dynamic Selection (STF) mechanisms. The GSLD module combines self-attention with convolution to capture large-scale static spatial features while preserving local details, solving the problem of dynamic information loss in single-frame image analysis. The DTA module integrates a bidirectional attention mechanism with positional encoding to achieve collaborative perception of short-term micro-expression fluctuations and long-term emotional trends. The STF module significantly improves the effectiveness of spatiotemporal feature fusion through a dual dynamic selection mechanism of channels and spatiotemporal features. The superiority of this invention is verified on a constructed depression video dataset. It not only overcomes the subjective bias of manual feature extraction and the shortcomings of insufficient receptive field in traditional deep models, but also achieves high-precision extraction of depression biomarkers implicit in patients' facial expressions through hierarchical dynamic selection of spatiotemporal features, providing a new method with both robustness and generalization ability for automated depression detection. Attached Figure Description

[0068] Figure 1 This is an overall architecture diagram of the FST-Net automatic depression diagnosis network in an embodiment of the present invention.

[0069] Figure 2 This is a structural diagram of the GSLD module in an embodiment of the present invention.

[0070] Figure 3 This is a diagram illustrating common methods for processing time information related to depression.

[0071] Figure 4 This is a structural diagram of the DTA module in an embodiment of the present invention.

[0072] Figure 5 This is a structural diagram of the STF module in an embodiment of the present invention.

[0073] Figure 6 Examples of visualizations of heatmaps at different times are provided; (a) is the attention heatmap of non-depressed samples correctly predicted by the model at different time segments, (b) is the attention heatmap of depressed samples correctly predicted by the model wearing masks at different time segments, and (c) is the attention heatmap of depressed samples correctly predicted by the model not wearing masks at different time segments. Detailed Implementation

[0074] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0075] like Figure 1 As shown, this embodiment of the invention provides a method for detecting depression by fusing spatiotemporal information, including the following steps:

[0076] Step 1, collect video image data of participants and pre-process to generate normalized dataset.

[0077] The data collection of this embodiment was conducted in the soundproof outpatient clinic of the Department of Psychiatry and Psychology, Affiliated Hospital of Guangdong Medical University. The conditions of the subjects in the case group all met the Diagnostic and Statistical Manual of Mental Disorders. The subjects in the control group were healthy people without a history of mental illness. This embodiment used the 24-item Hamilton Depression Scale (HAMD) to determine whether the subjects met the experimental criteria. The HAMD scale can be used for the auxiliary diagnosis of depression, evaluation of the severity of depressive symptoms, etc. During the experiment, two doctors guided the subjects simultaneously. The work of this embodiment was approved by the China Registered Clinical Trial Ethics Review Committee and strictly followed the ethical norms of the World Medical Association (Helsinki Declaration). Before the collection, all subjects were informed of the details of the experiment and confirmed their consent to participate.

[0078] The experimental subjects were 365 cases, including 173 patients with depression and 192 non-depression patients. The video was captured by a high-resolution camera. The camera was placed on the wall on both sides of the doctor and the angle of the camera was adjusted for each participant to ensure the best recording of facial features. The video frame rate was set to 30 fps. The complete video was removed from the doctor's speech and only the video clips of each patient answering questions were kept.

[0079] The dataset was divided into training set, validation set and test set using random stratified sampling, containing 217, 74 and 74 samples respectively, and the positive and negative sample ratio in each set is shown in Table 1.

[0080] Table 1 Distribution of positive and negative samples in dataset

[0081]

[0082] In order to eliminate the influence of useless background information on the experimental results, for the video data of each sample in the dataset, the Dlib was used to extract the facial key points in each frame of image, the affine transformation was used to align the face, and the image size was scaled to 224x224. In order to improve the generalization and robustness of the model, the brightness, contrast, saturation and hue of all frames in a video clip were randomly changed, and random horizontal flipping was performed; in the test stage, the cropped and aligned video frames were used without any enhancement. For a given test video, it was cropped into a 64-frame sub-video, and each segment was predicted to calculate the average value of the depression probability, thereby producing the final result of the video.

[0083] Step 2, build and train the depression recognition network FST-Net that fuses spatial and temporal information.

[0084] The depression recognition network FST-Net is designed and implemented under the PyTorch deep learning framework. The network model is trained using the SGD optimizer on a 16GB NVIDIA Tesla V100 with 16GB of video memory. The model training number is set to 200 rounds, the batch size is 8, the learning rate is 0.0001, the momentum is 0.9, and the weight decay is 0.0001. In order to improve the generalization ability of the model and prevent overfitting, the network weights are saved when the validation loss reaches the historical minimum value.

[0085] The cross-entropy loss function commonly used in classification tasks is selected. For binary classification tasks, the cross-entropy loss can be written as:

[0086]

[0087] where, is the true label (0: non-depression patient, or 1: depression patient), is the probability that the model predicts the patient as a depression patient.

[0088] Table 2 Depression classification confusion matrix

[0089]

[0090] The precision (Precision), recall (Recall), and F1 value are selected as the model performance evaluation indicators. In order to calculate these indicators, the depression classification confusion matrix shown in Table 2 is constructed.

[0091] The precision measures the proportion of actual positive classes in the model's prediction of positive classes, which reflects the accuracy of the model's prediction. The calculation formula is as follows:

[0092]

[0093] Recall measures the proportion of all actual positive samples that are correctly predicted as positive by the model. The calculation formula is as follows:

[0094]

[0095] F1 value is the harmonic mean of precision and recall, which balances between the two and provides a comprehensive performance measure considering both precision and recall. The calculation formula is as follows:

[0096]

[0097] To verify the effectiveness of FST-Net, it is compared with advanced methods in the research field. Since there are relatively few open-source video-based depression recognition methods, the present invention also selects some advanced methods widely recognized in the video understanding task as comparison models. The comparison methods mainly include two categories: video recognition-based depression detection methods and video understanding methods. The video recognition-based depression detection methods mainly include STA, MSN and MDN; the video understanding methods include the widely recognized classic model SlowFast, and the advanced methods VideoMAE2 and vid-TLDR. For these models, if the pre-training weight is provided, the pre-training model is used and fine-tuned. For SlowFast, VideoMAE2 and vid-TLDR, the number of input frames is set to 16, otherwise the pre-training weight cannot be used, and the above training process settings are used in the comparison experiments and ablation experiments in Tables 3 and 4.

[0098] Table 3 Comparison with advanced methods on benchmark datasets.

[0099]

[0100] The comparison experiment results of FST-Net and the comparison methods on the benchmark dataset are shown in Table 3, where the bold values are the optimal values, and the underlined values are the suboptimal values. From the results, it can be seen that first, the performance of FST-Net on the depression recognition task is significantly superior to other comparison methods. Specifically, the precision, recall and F1 value of FST-Net are 91.67%, 94.29% and 92.96% respectively, and two optimal values are obtained in the three evaluation indexes. Although the F1 value of FST-Net is slightly lower than that of vid-TLDR, the recall rate of vid-TLDR is only 82.86%. In summary, compared with other methods, FST-Net shows overall performance advantage. The video understanding models based on Transformer such as VideoMAE2 and vid-TLDR have F1 values of only 88.89% and 87.88%, and the possible reason is that they lack some inductive bias inherent to CNN, that is, the perception of locality, and need a large amount of data sets to achieve good results. The STA model with the same time attention mechanism processes time information by reducing the data in the vertical (height) and horizontal (width) directions. While the FST-Net model adopts a more comprehensive approach, which uses two independent branches to process the relationship between height and time and the relationship between width and time. This design enables FST-Net to capture and analyze the time features in the video more meticulously, thereby increasing the F1 value by 3.1% compared to the STA model. Compared with the MDN model, FST-Net provides a larger receptive field, making up for the weak ability of 3D CNN to capture long-distance spatial dependencies, and the F1 value is increased by 2.82%.

[0101] Secondly, in terms of model size, compared with vid-TLDR (64.97M) with the largest parameter size and MSN (696.65 GFlops) with the largest computation in the comparative methods, FST-Net achieved significant performance improvement with less parameter and computation. Compared with vid-TLDR, it increased by 11.43% in recall rate and 5.08% in F1 value; compared with MSN model, it led in three indicators, of which it increased by 5.56% in precision, 5.72% in recall and 5.64% in F1 value. In addition, compared with the 1.482M parameter amount of SlowFast (R50), although the computational complexity of FST-Net is significantly higher, this increased computational demand significantly improves the prediction performance of the model, making FST-Net superior to SlowFast (R50) in three evaluation indicators. Therefore, the advantage of the present application is to maintain a low computing cost while providing performance comparable to or even better than pre-trained models. This means that compared with the comparative methods, FST-Net is more in line with the needs of real application scenarios and is more suitable for clinical use.

[0102] In order to comprehensively evaluate the performance of the two modules GSLD and DTA in different combinations, a series of comparative experiments were conducted, involving six different application methods, and the experimental results are shown in Table 4.

[0103] When GSLD and DTA modules are used independently, the F1 values are 85.30% and 87.32% respectively; after combining the two modules in a sequential cascade manner, the F1 values are improved to varying degrees. First, using GSLD module first and then using DTA module, the F1 values are improved by 3.59% and 1.57% respectively compared with using the two modules alone. Similarly, using DTA module first and then using GSLD module, the F1 values are improved by 3.27% and 1.25% respectively. This shows that these two modules significantly improve the prediction performance of the model by focusing on the key spatial and temporal features in the video data. Specifically, the GSLD module can capture the subtle changes in the facial expressions of patients with depression by combining static global features and dynamic local features, which are often difficult for human observers to detect. The DTA module reveals the temporal dynamics of the behavior patterns of patients with depression by analyzing the changes of pixels in the vertical and horizontal directions over time.

[0104] Table 4 Performance comparison of different module combinations, + indicates sequential cascade use of modules, indicates parallel use of modules.

[0105]

[0106] In addition, the parallel use of the GSLD module and the DTA module has better performance, and compared with the sequential combination of the two modules, the F1 value is increased by 2.54% and 2.86%, respectively. When combined sequentially, the model may lose information in the process of feature transmission. The parallel structure can reduce this information loss, because each module can independently extract and utilize temporal or spatial information.

[0107] Further experiments show that using a better fusion strategy can further improve the performance of the model. When the outputs of the GSLD module and the DTA module are fused through the STF module, the prediction performance of the model reaches the best, with an F1 value of 92.96%, which is 1.53% higher than simply adding spatial and temporal features. The STF module fuses these features in a dynamic way, enabling the model to more comprehensively understand the spatiotemporal information in the video data. This spatiotemporal fusion method can improve performance better than using temporal or spatial features alone, because it can capture both the static features of the facial expressions of depressed patients and the temporal dynamics of their behaviors. In addition, compared with simple addition fusion, the STF module uses a more complex fusion strategy, enabling the model to more effectively utilize the spatiotemporal features in the data, thereby achieving more accurate depression prediction.

[0108] Figure 6 Some results of visualizing some consecutive frames of different samples using the general heat map visualization technique Grad-CAM are shown. Grad-CAM generates a heat map by analyzing the gradients of the network, highlighting the key areas in the image. By superimposing this heat map with the original image, we can visually see the parts that the model focuses on. In this embodiment, Grad-CAM is modified to adapt to the analysis needs of video data.

[0109] In Figure 6 (a), a sample judged as non-depression is shown; the heat map presents a lower activation intensity, indicating that the model does not detect significant features related to depression in these areas. In contrast, Figure 6 (b) and (c) show samples correctly identified as depression; in these samples, the heat map presents a higher activation intensity, especially around the eyes, nose, and mouth area. It is worth noting that even in Figure 6 (b), where the subject wears a mask, FST-Net can still accurately identify the key facial features. The heat map shows a higher activation intensity around the eyes, indicating that the model can extract effective features from limited facial information. Figure 6 (c) in (c) shows a depression sample without occlusion, and the heat map covers a wider facial area, including the forehead, nose, and mouth, showing that the model can comprehensively analyze facial expressions and capture the correlation of different parts.

[0110] Through the analysis of these visualization results, the following conclusions can be drawn: First, FST-Net exhibits excellent feature extraction capability. The model can accurately locate and analyze the key areas of the face, especially the eyes, nose, and mouth, which play a key role in the manifestation of depression. This precise feature positioning capability provides a solid foundation for accurate recognition of depressive symptoms. Second, FST-Net shows strong adaptability and robustness. Even in the case of partial face occlusion, such as the mask-wearing sample shown in (b) of FIG. 6, the model can still extract sufficient information from the visible facial area to make accurate judgments. Finally, FST-Net can capture dynamic features. By analyzing consecutive video frames, the model not only focuses on static facial features, but also captures subtle changes in facial expressions over time. This dynamic analysis capability enables the model to more comprehensively assess the emotional state of patients, providing richer and more accurate information than single static image analysis. Figure 6

[0111] Finally, it should be noted that the above examples are intended to illustrate the technical solutions of the present application and do not constitute any form of limitation on the present application. Those skilled in the art should fully understand that it is entirely feasible to modify the technical solutions described in the foregoing examples or to make equivalent replacements for any part or all of the technical features. These modifications or replacements, as long as they do not deviate from the protection scope determined by the claims of the present application, should be considered as reasonable extensions of the present application.​

Claims

1. A method for depression detection by fusing spatio-temporal information, characterized in that, The method comprises the following steps: S1, collecting video image data of participants and preprocessing to generate a normalized data set; S2, constructing a depression recognition network that fuses space-time information, which includes a static global dynamic local spatial attention module, a bidirectional time attention module and a space-time fusion module; The static global dynamic local spatial attention module processes the input image sequence. Global constant attention feature maps are obtained by using multi-head self-attention and convolutional normalization respectively. With local instantaneous features ,Will and Concatenation yields the feature vector The bidirectional temporal attention module runs in parallel with the static global dynamic local spatial attention module, processing the input image sequence. Parallel establishment of the interaction between the time and spatial dimensions yields time-highly correlated features. Time-width related features and will and Add to get the output The spatiotemporal fusion module processes the features obtained by the static global and dynamic local spatial attention modules. Features obtained with bidirectional temporal attention module To perform fusion, from feature maps and The system captures global temporal and spatial structure information respectively to obtain channel optimization features. With spatiotemporal weights Multiply them to obtain the final fused feature. ; S3, training the depression recognition network constructed in S2 using the normalized data set obtained in S1; S4, inputting the image of the patient to be predicted into the network model trained in S3 to obtain the depression prediction result of the patient. 2.The depression detection method of fusing space-time information according to claim 1, wherein, In S1, the specific process of generating a normalized data set comprises: S1.1, collecting facial features of participants and retaining video clips when answering questions; S1.2, for each participant's video data, the video is divided into an image sequence composed of several frame images; the facial key points in each frame image are extracted using Dlib, the face is aligned using affine transformation, and the image is scaled; S1.3, randomly change the brightness, contrast, saturation and hue of all frames in a video, and randomly perform horizontal flip to generate the final normalized data set. 3.The depression detection method of fusing space-time information according to claim 1, characterized in that, In the static global dynamic local spatial attention module, the feature vector The calculation formula is: The is a global constant attention feature map calculated by the multi-head attention mechanism, represented as: wherein, is a multi-head attention operation; denotes an average pooling operation; The Local temporal features of size 1x3x3 are extracted using a 1x3x3 convolutional layer followed by batch normalization and a ReLU activation function for non-linear transformation of the output, which is represented as: wherein, denotes a ReLU activation function; is batch normalization; denotes a 1x3x3 convolution to extract temporal features. 4.The method of claim 3, wherein, the global constant attention feature map The specific calculation process is as follows: to the input image average pooling operation along the time dimension and compress the time dimension to obtain a static expression feature : wherein, represents the number of channels, represents the time depth, , respectively represent the height and width of each frame of the input video. Subsequently, the static expression features are further extracted using multi-head self-attention mechanism MHSA The association of different regions in the process is as follows: wherein, is a weight matrix for synthesizing all attention heads; is the output of the th attention head, is the number of attention heads, is the product of the width and height of the image, The formula is: In the formula, is a static expression feature mapping different input projection weights; denotes an attention operation on the features therein.

5. The method of claim 1, wherein the method of depression detection fusing spatio-temporal information is characterized by, In the bidirectional time attention module, the time-high correlation feature The specific calculation process is as follows: For the input image sequence , the bidirectional temporal attention module firstly performs global average pooling operation and global max pooling operation on the horizontal dimension of the original horizontal aggregation vector respectively, and then splices the results along the horizontal dimension to obtain the original horizontal aggregation vector , and the process is represented as: wherein, denotes a global average pooling operation, denotes a global max pooling operation, denotes a concatenation operation along the horizontal dimension a concatenation operation; Then aggregate the original horizontal vector The dimensions are rearranged as Then, a 3D convolution with a kernel size of 1 is used to further increase the horizontal dimension. The information is aggregated to obtain horizontal aggregated features. : wherein, represents a rearrangement of the vector dimensions, represents a 3D convolutional layer with kernel size 1; Subsequently, the bidirectional time attention module compresses the dimension of the horizontally aggregated features to , and then uses a standard convolution with a kernel size of 3 to convolve the dimension with the dimension, which aggregates information on the time-high features; then, batch normalization and dimension expansion are performed, and the dimension is rearranged to , and the attention weight is obtained through a Sigmoid activation function. Finally, the original input is multiplied with to obtain the time-height correlation feature .

6. The method according to claim 5, wherein, In the bidirectional time attention module, the time-width correlation feature The specific calculation process is as follows: The bidirectional temporal attention module is used for input image sequence vertical dimension The global average pooling operation and the global maximum pooling operation are respectively performed, and then the results are spliced along the vertical dimension to obtain an original vertical aggregation vector The process is represented as: Then the dimension of the original vertical aggregation vector is rearranged as and a 3D convolution operation with a kernel size of 1 is performed to obtain the vertical aggregation feature : Subsequently, the bidirectional temporal attention module compresses the dimension of the vertically aggregated features to , and then uses a standard convolution with a kernel size of 3 to perform convolution on the dimension and the dimension, and performs batch normalization, then expands and rearranges the dimension to , and obtains the attention weight : Finally, the original input is multiplied with to obtain the time-width correlation feature .

7. The method according to claim 1, wherein, In the spatio-temporal fusion module, the fusion feature The specific implementation process is as follows: The spatiotemporal fusion module first combines the feature maps and The channels are concatenated; then global channel weights are obtained through global pooling, convolution operations, and the Sigmoid activation function. : wherein, denotes a channel dimension performing a concatenation operation, denotes a fusion feature, denotes a global pooling operation; Utilizing global channel weights On fused features Calibration; subsequently, utilizing 1x1x1 convolution to select feature maps according to the importance of the feature maps; global channel weights Retain important features while discarding less informative features , obtaining channel-optimized features : To model the spatio-temporal interdependence between local feature maps in depth, the spatio-temporal fusion module captures the global temporal and spatial structure information from the feature maps and respectively by standard convolution operation with kernel size 1x1x1 and Sigmoid activation function, whose sum is the spatio-temporal weight : Finally, the spatiotemporal weight with the channel optimization feature is multiplied: wherein, is a spatiotemporal fusion feature.

8. The method of claim 1, wherein the method comprises: In step S3, the SGD optimizer is used for training. A small batch of samples is randomly sampled from the training data, and the samples are processed through a depression recognition network to obtain fused features. The final sample prediction result is obtained after applying the activation function. and through Calculate the gradient of the loss function under the current network model parameters, and update the learning parameters according to the gradient direction and learning rate. Repeat the above training process, traversing the entire training dataset, until the loss function converges to a stable value.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor; characterized in that, The processor executes the computer program, so that the electronic device executes the depression detection method fusing space-time information according to any one of claims 1-8.

10. A storage medium comprising a computer program, which, when executed on an electronic device, causes the electronic device to execute a depression detection method fusing space-time information according to any one of claims 1-8.

Images