Space-time fusion network-based depression prediction method

Abstract

The invention belongs to the field of intelligent medical treatment, and particularly relates to a depression prediction method based on a space-time fusion network. According to the invention, a space-time attention mechanism is introduced, so that the spatial-temporal feature fusion capability of facial expressions is enhanced; a strategy based on attention vector fusion is provided, and the adaptability of space and time features is improved; the STA module is stacked to be connected with the residual error, so that the problem of long-time dependence is effectively solved; by adopting a visual analysis technology, the interpretability of the model is improved, and the relationship between a facial area and depression is disclosed; the defects of the existing method in the aspects of spatial-temporal feature fusion, long-time dependence on modeling and interpretability are overcome. When a depression identification task is processed, the method shows strong cross-task migration ability, outstanding space-time fusion ability and good robustness, and provides an efficient and explainable intelligent analysis solution for early screening of depression.

Description

Technical Field

[0001] This invention belongs to the field of smart healthcare, specifically relating to a method for predicting depression based on a spatiotemporal fusion network. Background Technology

[0002] In recent years, artificial intelligence (AI) technology has made significant progress in the application of mental health, particularly in the identification of depression. Traditional methods for depression identification rely on psychological interviews and questionnaires, while modern technologies utilize multimodal data such as facial expressions, voice, and physiological signals for analysis. Deep learning-based methods have achieved major breakthroughs, especially in facial video analysis. These methods effectively identify depressive states by extracting facial features and combining them with techniques such as sentiment analysis. Convolutional Neural Networks (CNNs) and Visual Transformers (ViTs) are currently the mainstream models widely used in video sentiment analysis. CNNs perform well in static image analysis, but their limited ability to extract local features makes it difficult to effectively capture the temporal changes in facial expressions. While ViTs can model long-range dependencies through a global self-attention mechanism, their application in facial video analysis still faces challenges such as high computational complexity and large data requirements. Furthermore, the inadequacy of spatiotemporal feature modeling remains a technical challenge for facial video depression identification.

[0003] Current research mainly focuses on the following types of methods:

[0004] (1) Convolutional Neural Network (CNN) series models:

[0005] For example, Zhu et al. (2018) proposed a two-stream CNN-based architecture to process facial images and optical flow information separately to identify depression. However, these methods typically only process single-frame or short-term dynamic features, ignoring temporal changes in facial expressions and failing to comprehensively capture the long-term facial changes associated with depression.

[0006] (2) A series of models based on 3D CNN:

[0007] For example, De Melo et al. (2019) used 3D convolutional neural networks (CNNs) to extract long-time frame features from videos. Although 3D CNNs can effectively extract temporal information from videos, the translation invariance of convolution and pooling operations lead to the dilution of spatial-temporal information, making it difficult to effectively capture key features in specific temporal or spatial regions.

[0008] (3) A series of models based on attention mechanisms:

[0009] In recent years, attention mechanisms have been introduced into the identification of depression in facial videos to enhance the adaptive weighting of features. The Deep Local GlobalAttention CNN proposed by He et al. (2021) has shown good performance in learning global and local representations. However, these methods often lack sufficient temporal information and fail to effectively combine spatial and temporal features, resulting in limited performance improvements.

[0010] Based on the above research, existing facial video methods for identifying depression still have the following shortcomings:

[0011] 1. Lack of effective fusion of spatiotemporal features: Most existing methods fail to adequately integrate spatial and temporal features in facial videos, especially when capturing dynamic features of facial expressions changing over time, exhibiting significant limitations. For example, CNNs typically rely on translation invariance and pooling operations, which lead to the loss of spatiotemporal information and dilution of feature representations.

[0012] 2. Imperfections in Spatial and Temporal Attention Mechanisms: Although some methods employ attention mechanisms to weight different features, these methods often fail to effectively capture key spatial and temporal information separately. Spatial-temporal feature fusion strategies often fail to preserve their original structure, leading to information loss when extracting facial expressions.

[0013] 3. Insufficient capture of long-term dependence: Existing models usually focus on short-term dynamic feature extraction and have difficulty capturing long-term facial expression changes, resulting in the inability to accurately identify chronic facial features of depression.

[0014] These problems are particularly prominent in the identification of depression in facial video, which severely limits the practical application value of the model. Summary of the Invention

[0015] To address the shortcomings of existing methods for depression identification, this invention proposes a depression identification network based on a space-time attention mechanism (STA-DRN). By introducing the space-time attention mechanism (STA), an attention vector fusion strategy, and residual connections, STA-DRN effectively solves the following technical problems:

[0016] 1. Introduce a space-time attention mechanism to enhance the spatiotemporal feature fusion capability of facial expressions;

[0017] 2. A strategy based on attention vector fusion is proposed to improve the adaptability of spatial and temporal features;

[0018] 3. By stacking STA modules and connecting them with residuals, long-term dependency issues can be effectively addressed;

[0019] 4. Employing visualization analysis techniques enhances model interpretability and reveals the relationship between facial regions and depression;

[0020] 5. Improve the accuracy of depression identification and address the shortcomings of existing methods in terms of spatiotemporal feature fusion, long-term dependency modeling, and interpretability.

[0021] This invention provides a method for predicting depression based on a spatiotemporal fusion network, comprising the following steps:

[0022] After preprocessing the continuous video, it is input into a pre-trained depression prediction model to obtain depression prediction results;

[0023] The depression prediction model comprises a feature extraction module, a spatial-temporal attention fusion module, and a classification output module connected in sequence; wherein, there is more than one spatial-temporal attention fusion module, and the spatial-temporal attention fusion modules are connected in sequence.

[0024] The specific structure of the space-time attention fusion module includes:

[0025] The spatial feature extraction module and the temporal feature extraction module are used to receive input data and extract spatial and temporal features, respectively.

[0026] The fusion module is used to fuse the outputs of the spatial feature extraction module and the temporal feature extraction module, and the fusion result is used as the output of the spatial-temporal attention fusion module;

[0027] The classification output module is used to output the classification result of whether or not the individual belongs to depression.

[0028] Preferably, the preprocessing includes frame extraction, resizing, and normalization.

[0029] Preferably, the connection method of the space-time attention fusion module is residual connection.

[0030] Preferably, the feature extraction module is one of 3D ResNet, I3D, R(2+1)D, or SlowFast structures.

[0031] Preferably, the number of the space-time attention fusion modules is two.

[0032] Preferably, the spatial feature extraction module is one of Spatial Attention, CBAM, SE, or self-attention.

[0033] Preferably, the time feature extraction module is one of Temporal Attention, LSTM, TSM or Transformer-Encoder.

[0034] Preferably, the fusion method of the fusion module is one of weighted summation, concatenation + convolution, or Transformer-based fusion.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0036] Beneficial effects:

[0037] 1) By introducing a spatial-temporal attention mechanism (STA), the spatiotemporal feature extraction of facial expressions is improved, and the sensitivity to dynamic facial features is enhanced;

[0038] 2) By employing an attention vector fusion strategy, the adaptability of spatial and temporal features is enhanced, thereby improving the recognition accuracy of facial regions related to depression.

[0039] 3) By stacking STA modules and connecting them to residuals, the modeling capability for long-term dependencies is improved, and continuous changes in facial expressions are captured;

[0040] 4) Improve accuracy in depression identification tasks and achieve competitive performance on psychology datasets. Attached Figure Description

[0041] Figure 1 This is a model structure diagram in an embodiment of the present invention;

[0042] Figure 2 This is a comparison diagram of the effects in the embodiments of the present invention. Detailed Implementation

[0043] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0044] like Figures 1 to 2 As shown, a depression prediction method based on spatiotemporal fusion networks includes the following steps:

[0045] After preprocessing the continuous video, it is input into a pre-trained depression prediction model (STA-DRN model) to obtain depression prediction results; this can serve as a reference for doctors to diagnose depression.

[0046] The preprocessing includes frame extraction, resizing, and normalization.

[0047] In this embodiment, the preprocessing step extracts each video frame, unifies its size to 224×224, and aligns it using the Dlib tool to remove background interference and ensure the stability of the facial images. Preprocessing continuous video is a common technique in the art and will not be elaborated upon here.

[0048] The depression prediction model comprises a feature extraction module, a spatial-temporal attention fusion module (STA module), and a classification output module connected in sequence; wherein, there is more than one spatial-temporal attention fusion module, and the spatial-temporal attention fusion modules are connected in sequence.

[0049] The spatial-temporal attention fusion module uses residual connections for its connections.

[0050] The specific structure of the space-time attention fusion module includes:

[0051] The spatial feature extraction module and the temporal feature extraction module are used to receive input data and extract spatial and temporal features, respectively.

[0052] The fusion module is used to fuse the outputs of the spatial feature extraction module and the temporal feature extraction module, and the fusion result is used as the output of the spatial-temporal attention fusion module;

[0053] The classification output module includes an adaptive pooling layer and a fully connected layer, which are used to output the classification result of whether or not the individual belongs to depression.

[0054] like Figure 1 As shown, in this embodiment, the feature extraction module is a 3D ResNet; in some possible implementations, the feature extraction module may also be an I3D, R(2+1)D, or SlowFast structure.

[0055] In this embodiment, there are two spatial-temporal attention fusion modules, which are connected sequentially.

[0056] In this embodiment, the spatial feature extraction module is Spatial Attention; in some possible implementations, the spatial feature extraction module may also be CBAM, SE, or self-attention.

[0057] In this embodiment, the temporal feature extraction module is Temporal Attention; in some possible implementations, the temporal feature extraction module may also be LSTM, TSM or Transformer-Encoder.

[0058] In this embodiment, the fusion method of the fusion module is weighted summation; in some possible implementations, the fusion method of the fusion module can also be any one of splicing + convolution or Transformer-based fusion.

[0059] Example of effect verification:

[0060] like Figure 2 As shown, in order to verify the effectiveness of the STA-DRN model, this embodiment conducted a comparative experiment with two other models on the same psychology dataset.

[0061] (1) Experimental comparison setup

[0062] Comparative Experiment 1 uses an LSTM-based sequence classification model, aiming to detect depression by analyzing the time-series features of video frames; Comparative Experiment 2 uses a ResNet-18-based binary image classification model, aiming to determine whether an individual may have depression by analyzing facial image features.

[0063] (2) Performance comparison results

[0064] As can be seen, STA-DRN outperforms both Control Experiment 1 and Control Experiment 2 in terms of test set accuracy, recall, and F1 score. The detailed analysis is as follows:

[0065] Test set accuracy: The accuracy of STA-DRN was 77.50%, which is a significant improvement compared to Comparative Experiment 1 (70.00%) and Comparative Experiment 2 (65.00%), indicating that STA-DRN has superior overall performance in the depression recognition task.

[0066] Precision: Although the precision of the control experiment was higher (83.33%) than that of the control experiment 1, the STA-DRN performed better in terms of recall and F1 score, especially in terms of recall, which reached 90.00%, significantly higher than the 50.00% of the control experiment 1.

[0067] Recall: The recall rate of STA-DRN reached 90.00%, indicating that the model has stronger sensitivity in identifying patients with depression, and performs better than Comparative Experiment 1 (50.00%) and Comparative Experiment 2 (80.00%).

[0068] F1 score: The F1 score of STA-DRN is 80.00%, which is better than control experiment 1 (62.50%) and control experiment 2 (69.57%) when considering the balance between precision and recall.

[0069] (3) Performance Improvement Analysis

[0070] By introducing a spatiotemporal attention mechanism (STA), STA-DRN can capture global and local spatiotemporal features of facial expressions, enhancing the model's sensitivity to facial expressions related to depression. Furthermore, combining ResNet-style stacked STA modules further improves the model's performance. Therefore, the STA-DRN model proposed in this invention can provide higher accuracy, precision, and recall in depression identification tasks, maintaining stable superior performance across different datasets and tasks.

[0071] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for predicting depression based on spatiotemporal fusion networks, characterized in that, Includes the following steps: After preprocessing the continuous video, it is input into a pre-trained depression prediction model to obtain depression prediction results; The depression prediction model comprises a feature extraction module, a spatial-temporal attention fusion module, and a classification output module connected in sequence; wherein, there is more than one spatial-temporal attention fusion module, and the spatial-temporal attention fusion modules are connected in sequence. The specific structure of the space-time attention fusion module includes: The spatial feature extraction module and the temporal feature extraction module are used to receive input data and extract spatial and temporal features, respectively. The fusion module is used to fuse the outputs of the spatial feature extraction module and the temporal feature extraction module, and the fusion result is used as the output of the spatial-temporal attention fusion module; The classification output module is used to output the classification result of whether or not the individual belongs to depression.

2. The method for predicting depression based on a spatiotemporal fusion network according to claim 1, characterized in that, The preprocessing includes frame extraction, resizing, and normalization.

3. The method for predicting depression based on a spatiotemporal fusion network according to claim 1, characterized in that, The connection method of the space-time attention fusion module is residual connection.

4. The method for predicting depression based on a spatiotemporal fusion network according to claim 1, characterized in that, The feature extraction module is one of the following structures: 3D ResNet, I3D, R(2+1)D, or SlowFast.

5. The method for predicting depression based on a spatiotemporal fusion network according to claim 1, characterized in that, The number of the space-time attention fusion modules is two.

6. The method for predicting depression based on a spatiotemporal fusion network according to claim 1, characterized in that, The spatial feature extraction module is one of Spatial Attention, CBAM, SE, or self-attention.

7. The method for predicting depression based on a spatiotemporal fusion network according to claim 1, characterized in that, The time feature extraction module is one of Temporal Attention, LSTM, TSM, or Transformer-Encoder.

8. The method for predicting depression based on a spatiotemporal fusion network according to claim 1, characterized in that, The fusion module uses one of the following fusion methods: weighted summation, concatenation + convolution, or Transformer-based fusion.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 8.

Images