Depression detection method and system based on long-short time multi-modal time series modeling and medium

Abstract

The application discloses a depression detection method and system based on long-short time multi-modal time sequence modeling and a medium, relates to the technical field of artificial intelligence, and comprises the following steps: acquiring audio features and video features of a to-be-detected video, and cutting the to-be-detected video into multiple short-time segments; obtaining segment-level fusion representations by respectively encoding, cross-modal attention and then fusing the audio features and the video features of each short-time segment; decoupling the segment-level fusion representations into depression-related representations and depression-irrelevant representations, training in combination with a classification loss, a representation separation constraint and a reconstruction constraint, and extracting depression-related representations of each short-time segment; constructing a graph structure based on each short-time segment and the depression-related representations, updating node representations through a graph attention network, aggregating, classifying a long video global representation obtained, and outputting a depression detection result; and the method takes into account local discriminant sensitivity and global trend stability, and effectively improves recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, and medium for detecting depression using long- and short-time multimodal temporal modeling. Background Technology

[0002] Depression is a common mental disorder, often characterized by persistent low mood and diminished interest and pleasure. Its incidence continues to rise, placing a heavy burden on individuals, families, and society. Traditional depression assessments primarily rely on scale scores and physician interviews, which suffer from high labor costs, strong subjectivity, and difficulty in large-scale, continuous monitoring, failing to meet the real-world needs for efficient, objective, and automated assisted screening.

[0003] With the development of affective computing and deep learning technologies, automatic analysis of depressive states based on multimodal behavioral signals such as audio and video has gradually become an important research direction for intelligent mental health auxiliary assessment. Compared with single-modal methods, multimodal methods can simultaneously utilize multiple sources of cues such as speech rhythm, pauses, energy changes, facial expressions, eye contact, and head posture to more comprehensively reflect an individual's emotional and behavioral state.

[0004] However, existing technologies still have significant shortcomings in long video scenarios: First, depression-related evidence is usually only distributed in a small number of local segments. If the entire long sequence is directly encoded and aggregated in a unified manner, key discriminative clues are easily diluted by the global averaging process. Second, long-range dependencies across segments are difficult to fully model using simple convolution or recurrent structures. Third, multimodal fusion features are often mixed with interfering factors that have no direct correlation with the depressive state, such as individual expression habits, scene noise, and fluctuations in acquisition quality, thereby reducing the stability and generalization ability of the model. Summary of the Invention

[0005] Based on the technical problems existing in the background technology, this invention proposes a method, system and medium for depression detection based on long and short time multimodal temporal modeling, which takes into account both local discrimination sensitivity and global trend stability, and effectively improves the recognition accuracy.

[0006] The depression detection method based on long- and short-time multimodal temporal modeling proposed in this invention includes: The audio and video features of the video to be detected are obtained and segmented into multiple short segments; The audio and video features of each short segment are encoded and then fused together through cross-modal attention to obtain a segment-level fused representation; The segment-level fusion representation is decoupled into depression-related representations and depression-unrelated representations by a representation decoupling module, and trained based on classification loss, representation separation constraints and reconstruction constraints to extract depression-related representations for each short segment. Using each short video segment as a node and the depression-related representation of each short video segment as the initial representation of the node, a graph structure containing local temporal edges and global connection edges is constructed in chronological order. The node representations are updated and aggregated through a graph attention network. The obtained global representation of the long video is then classified, and the depression detection result is output.

[0007] Furthermore, the audio and video features are respectively encoded and fused through cross-modal attention to obtain a segment-level fused representation, specifically as follows: Audio features are encoded using an audio encoder to obtain audio modality-coded features; Video features are encoded using a video encoder to obtain video modality-coded features; Audio modality coding features and video modality coding features are each dimensionality-reduced using a convolutional neural network and then mapped to the same embedding space. In the embedding space, we construct the attention of audio to video and the attention of video to audio. We then perform residual fusion of the audio-to-video attention result with the dimensionality-reduced audio modal coding features and residual fusion of the video-to-audio attention result with the dimensionality-reduced video modal coding features. The two modal features after residual fusion are pooled separately and then concatenated to obtain a fragment-level fused representation.

[0008] Furthermore, the segment-level fusion representation is decoupled into depression-related and depression-independent representations through the representation decoupling module, and trained based on classification loss, representation separation constraints, and reconstruction constraints to extract depression-related representations for each short segment, specifically as follows: The fragment-level fusion representation is input into a parallel set depression-related convolutional branch and a depression-independent convolutional branch to obtain depression-related representations and depression-independent representations. The separation and decoupling between depression-related representations and depression-independent representations are constrained by representation separation constraints. A fully connected classifier is used to classify and predict depression-related features in each short time segment, and the fully connected classifier and depression-related convolutional branch are optimized based on the classification loss. The classification loss, representation separation constraint, and reconstruction constraint are combined to form the optimization objective function, which is used to update the representation decoupling module.

[0009] Furthermore, optimize the objective function. as follows: ; ; ; ; in, For classifying losses, To characterize the separation constraints, To divide and reconstruct constraints, and Indicates the weighting coefficient. The total number of short segments. For indexing short time segments, A true label of depression for a short period of time. For the first The prediction results of the classifier output from the depression-related characteristics of a short time segment. These are symptoms associated with depression. These are symptoms unrelated to depression. For the fragment-level fusion feature, the first One element, Represents the reconstruction of the first One element, This indicates the total number of characteristic elements.

[0010] Furthermore, the construction of a graph structure containing local temporal edges and global connectivity edges in chronological order is specifically as follows: An edge will be established between two nodes that satisfy the adjacency matrix in time, thus constructing a local temporal edge; Connect any two nodes with an edge to construct a fully connected edge.

[0011] Furthermore, the adjacency matrix is ​​specifically as follows: ; in, For the first The node and the first The adjacency matrix of nodes in time It represents the local time neighborhood.

[0012] Furthermore, the global representation of the long video is obtained by pooling all updated node representations.

[0013] Furthermore, in the classification of the long video global representation, the classification uses a fully connected layer to output the depression probability or category.

[0014] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0015] A computer-readable storage medium storing a plurality of computer programs for being invoked by a processor and executing the method described above.

[0016] The advantages of the depression detection method, system, and medium provided by this invention, which utilizes long-term and short-term multimodal temporal modeling, are as follows: At the short-term level, it extracts fragment-level evidence from long videos, and through the fusion of multimodal interactions within fragments and the separation of depression-related representations, it can more sensitively capture key local clues. At the long-term level, it simultaneously models local temporal relationships and global cross-fragment associations for fragment sequences, and utilizes graph attention mechanisms to achieve adaptive propagation and aggregation of evidence, enhancing the ability to characterize long-range dependencies and overall behavioral trends in complex long sequences. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of the present invention, wherein GAT is a graph attention network, GAP is global average pooling, the Global GAT branch is a global graph attention branch based on a fully connected graph, used to model the global association between short time segments; the Local GAT branch is a local graph attention branch based on a temporal adjacency graph, used to model the local temporal relationship between short time segments; Figure 2 A schematic diagram representing the structure of the decoupling module; Figure 3 This is a schematic diagram of the graph attention network GAT. Figure 4 A scatter plot of the characteristic distribution; Figure 5 This is a probability distribution diagram of the depression detection model. Figure 6 This is a diagram illustrating the changes in accuracy during the training process. Figure 7 This is a diagram illustrating the changes in F1 score during the training process; Figure 8 A diagram illustrating the change in loss values ​​as the objective function is optimized during the training process. Detailed Implementation

[0018] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0019] This embodiment sets up a depression detection model that uses the visual and auditory features of the subjects' facial expression videos as the basis for result classification. The depression detection model includes a multimodal fusion module, a representation decoupling module, and a graph attention network.

[0020] like Figures 1 to 8As shown, the depression detection method based on long-short-time multimodal temporal modeling proposed in this invention includes the following steps: Step 1: Obtain the audio and video features of the video to be detected, and segment it into multiple short segments; Step 2, Multimodal Fusion Module: The audio and video features of each short segment are encoded and then fused together through cross-modal attention to obtain a segment-level fused representation; Step 3: Decouple the fragment-level fusion representation into depression-related representations and depression-unrelated representations through the representation decoupling module. Train the representations using classification loss, representation separation constraints, and reconstruction constraints to extract depression-related representations for each short-term fragment. Step 4: Graph Attention Network: Using each short video segment as a node and the depression-related representation of each short video segment as the initial representation of the node, a graph structure containing local temporal edges and global connection edges is constructed in chronological order. The node representations are updated and aggregated through the graph attention network. The obtained global representation of the long video is then classified, and the depression detection result is output.

[0021] This depression detection model uses a hierarchical detection framework consisting of short-term segment-level cross-modal interaction fusion, depression-related characterization extraction, and long-term graph structure temporal modeling to detect depression. It also takes into account local discrimination sensitivity and global trend stability, effectively improving the recognition accuracy.

[0022] This embodiment extracts fragment-level evidence from long videos at the short-term level. By fusing multimodal interactions within fragments and separating depression-related representations, it can more sensitively capture key local clues. At the long-term level, it simultaneously models local temporal relationships and global cross-fragment associations for fragment sequences and uses graph attention mechanisms to achieve adaptive propagation and aggregation of evidence, enhancing the ability to characterize long-range dependencies and overall behavioral trends in complex long sequences.

[0023] I. Construction of audio and video features; Prepare long video samples tagged with depression, and extract audio and video features from each long video sample; segment each long video according to a preset length to obtain... Each short segment, including corresponding audio and video features, forms either a training dataset or a test dataset.

[0024] Let the first The audio and video features of each short segment are represented as follows: ; in, Indicates the first Audio characteristics of a short segment, Indicates the first A short video segment's features, Indicates the duration of a short segment. Represents the audio feature dimension. The dimension representing video features can be set. and The values ​​are 25 and 136.

[0025] II. Multimodal fusion module; The audio and video features of each short segment are input into the dual-stream coding module to obtain their respective temporal coding results. Then, through convolutional dimensionality reduction, bidirectional cross-modal attention interaction, and residual connections, a segment-level fused representation is obtained.

[0026] The dual-stream coding module includes an audio encoder and a video encoder. This embodiment does not limit the specific network structures of the two encoders; any neural network module capable of feature extraction and encoding of temporal signals is acceptable. For example, the two encoders could employ LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), or Transformer encoders. In the specific implementation of this embodiment, the audio encoder and video encoder use Transformer encoders and participate in parameter updates during the training process of the multimodal fusion module.

[0027] Audio features are encoded using an audio encoder to obtain audio modality-coded features; video features are encoded using a video encoder to obtain video modality-coded features.

[0028] ; in, Indicates an audio encoder. Indicates video encoder, and These represent audio modal coding features and video modal coding features, respectively.

[0029] Then, the audio modality coding features and video modality coding features are respectively mapped to the same embedding space after dimensionality reduction by a convolutional neural network; the main functions are as shown in (a1) to (a3): (a1) Dimensional uniformity: The original dimensions of audio features and video features are different, which are 25 and 136 dimensions respectively in this embodiment. After being encoded by their respective encoders, the two types of features are mapped to 256 dimensions, and then further reduced to the same 64-dimensional representation by a convolutional neural network. This satisfies the dimension matching requirements of matrix operations between queries, keys and values ​​in subsequent cross-modal attention calculations, and provides a unified feature representation basis for bidirectional cross-modal interaction.

[0030] (a2) Semantic alignment: Audio and video belong to heterogeneous modalities, and their underlying feature distribution and semantic expression differ. Mapping to a common embedding space can semantically align the features of different modalities, making audio cues (such as speech rate and energy) and video cues (such as facial expressions and eye contact) in the same depressive state comparable in space, thereby effectively measuring cross-modal similarity.

[0031] (a3) Promote cross-modal interaction: In a unified 64-dimensional representation space, query-key-value (QKV) attention of audio to video and attention of video to audio can be constructed to achieve bidirectional fusion of information from the two modalities and improve the discriminative ability of the fused representation.

[0032] (a4) Reduce computational complexity: By performing dimensionality reduction on the encoded features through convolutional neural networks, the feature dimension can be reduced while retaining the main discriminative information, thereby reducing the parameter scale and computational overhead in subsequent cross-modal attention calculations, improving the efficiency of model training and inference, and mitigating the interference of redundant information caused by direct interaction of high-dimensional features.

[0033] In the embedding space, we construct the attention of audio to video and the attention of video to audio. We then perform residual fusion of the audio-to-video attention result with the dimensionality-reduced audio modal coding features and residual fusion of the video-to-audio attention result with the dimensionality-reduced video modal coding features.

[0034] ; ; in, For the first Attention results of audio features of short time segments to video features. for Attention results of video features on audio features in short time segments For the first The audio features of a short time segment (after dimensionality reduction) are used as the query. For the first The query uses the video features (after dimensionality reduction) of a short time segment. For the first The video features of a short time segment (after dimensionality reduction) are used as keys. For the first The audio features of a short time segment (after dimensionality reduction) are used as keys. For the first The video features (after dimensionality reduction) of a short time segment are used as values. For the first The audio features of a short time segment (after dimensionality reduction) are used as values. For feature dimensions.

[0035] Next, the cross-modal interaction results are residually fused with the original modal features to obtain: ; ; in, To be and Modal features after residual fusion To be and Modal characteristics after residual fusion.

[0036] Finally, the two modal features after residual fusion are pooled separately and then concatenated to obtain a fragment-level fused representation.

[0037] ; in, For the first Fragment-level fusion representation of short time segments, For pooling operations, This indicates a splicing operation. In the specific implementation of this embodiment, the pooled audio and video representations are both 32-dimensional, and after splicing, they form a 64-dimensional segment-level fused representation.

[0038] III. Characterization Decoupling Module; The fragment-level fusion representation obtained above is input into the representation decoupling module to extract depression-related representations and depression-irrelevant representations respectively. Then, through classification loss, representation separation constraint and reconstruction constraint, the depression detection model retains the discriminative information related to the depressive state and outputs depression-related representations of short-term fragments.

[0039] First, the fragment-level fused representation is input into a depression-related convolutional branch set in parallel. Convolutional branches unrelated to depression The depression-related and depression-independent representations are obtained. Both the depression-related and depression-independent convolutional branches are constructed using a multi-layered cascaded one-dimensional convolutional structure. The layers are connected in the order of convolutional layer, normalization layer, and activation layer. The depression-related convolutional branch is used to extract depression-related representations, and the depression-independent convolutional branch is used to extract depression-independent representations. The separation and decoupling between depression-related and depression-independent representations are constrained by representation separation constraints.

[0040] In the specific implementation of this embodiment, both the depression-related and depression-independent convolutional branches include three cascaded one-dimensional convolutional layers, each with a kernel size of 1 and output channels of 512, 128, and 32 respectively. A normalization layer and an activation layer are placed after each adjacent convolutional layer; the normalization layer is a Group Normalization layer, and the activation layer is a ReLU activation layer. Through this structure, the input fragment-level fusion features are progressively mapped to a 32-dimensional low-dimensional representation.

[0041] Therefore, depression-related and depression-unrelated characteristics are represented as follows: ; ; in, For the first A short-term segment of depression-related symptoms, For the first A short-term, unrelated symptom of depression.

[0042] Then, using a linear classifier The depression-related characteristics of each short time segment are classified and predicted to obtain segment-level intermediate prediction results. The linear classifier and the depression-related convolutional branch are then optimized using classification loss. .

[0043] in, For the first The segment-level intermediate prediction results are output by a linear classifier based on the depression-related representations of each short-term segment. The segment-level intermediate prediction results are used to constrain the depression-related representations to retain the discriminative information related to the depressive state during the training process of the representation decoupling module. Finally, the depression detection result is output by the subsequent graph attention network after graph-level aggregation based on the depression-related representations of each short-term segment.

[0044] Its classification loss for: ; in, The total number of short segments. For indexing short time segments, This is a true label for short-term, fragmented depression.

[0045] To constrain the separation between depression-related and depression-irrelevant representations, a representation separation constraint is set. for: ; in, These are symptoms associated with depression. This represents depression-irrelevant representations. The representation separation constraint is used to reduce the correlation between depression-related and depression-irrelevant representations, enabling both branches to learn mutually separate representation spaces.

[0046] Specifically, this characterization separates constraints. This minimizes the inner product (correlation) between the outputs of the two branches (i.e., the depression-related convolutional branch and the depression-independent convolutional branch). If the features of the outputs of the two branches are highly similar, The similarity would be very large, thus penalizing such similarity. Therefore, the training process drives the two branches to learn non-overlapping representation spaces: one space stores depression-discriminating information, and the other space stores depression-irrelevant information.

[0047] Reconstruction constraints The expression for the difference between the fragment-level fused features and the reconstructed representation used to constrain the input is: ; in, In the fragment-level fusion feature, the first One element, Represents the reconstruction of the first One element, The total number of feature elements is represented. The reconstructed representation refers to the process of fusing depression-related and depression-independent representations and inputting the resulting data into a reconstruction encoder. This is then mapped back to a fragment-level fusion feature space through multi-layer one-dimensional convolutions to obtain the reconstructed representation. Reconstruction constraints are used to ensure that the reconstructed representation remains consistent with the fragment-level fusion features of the input representation decoupling module, guaranteeing that the two types of representations retain the effective information from the original features after decoupling.

[0048] Finally, the classification loss, representation separation constraint, and reconstruction constraint are combined to form the optimization objective function. Based on this, the decoupling module is updated: ; in, , and These are weighting coefficients, used to adjust the proportions of classification loss, characterization separation constraint, and reconstruction constraint in the optimization objective function, respectively. In the specific implementation of this embodiment, =1.0, =0.075, =0.025.

[0049] Through the aforementioned representation decoupling module, the depression-related convolutional branch retains discriminative information related to the depressive state under the drive of classification loss; the depression-independent convolutional branch learns information that is relatively unrelated to the classification task but helps to reconstruct the input features under the combined effect of representation separation constraints and reconstruction constraints; the reconstruction constraints ensure that the effective information in the fragment-level fusion features can still be retained after the two branches are decoupled, thereby improving the discriminativeness and stability of the initial representation of nodes in the subsequent graph attention network.

[0050] As one embodiment, the structure of the depression-related convolutional branch includes three sequentially connected one-dimensional convolutional layers: one-dimensional convolutional layer 1-1, one-dimensional convolutional layer 1-2, and one-dimensional convolutional layer 1-3. One-dimensional convolutional layers 1-1 and 1-2 are followed by a GroupNorm normalization layer and a ReLU activation layer, while one-dimensional convolutional layer 1-3 is followed by a GroupNorm normalization layer and a ReLU activation layer. One-dimensional convolutional layer 1-1 maps the input fragment-level fused features to 512-dimensional features, one-dimensional convolutional layer 1-2 maps the 512-dimensional features to 128-dimensional features, and one-dimensional convolutional layer 1-3 serves as the output layer, mapping the 128-dimensional features to 32-dimensional features, which are then used as the depression-related representation.

[0051] For the depression-independent convolutional branch, its structure includes three sequentially connected one-dimensional convolutional layers: one-dimensional convolutional layer 2-1, one-dimensional convolutional layer 2-2, and one-dimensional convolutional layer 2-3. One-dimensional convolutional layers 2-1 and 2-2 are followed by a GroupNorm normalization layer and a ReLU activation layer, while one-dimensional convolutional layer 2-3 is followed by another GroupNorm normalization layer and a ReLU activation layer. One-dimensional convolutional layer 2-1 maps the input fragment-level fused features to 512-dimensional features, one-dimensional convolutional layer 2-2 maps the 512-dimensional features to 128-dimensional features, and one-dimensional convolutional layer 2-3 serves as the output layer, mapping the 128-dimensional features to 32-dimensional features, which are then used as the depression-independent representation.

[0052] Depression-related convolution branches suffer from classification loss This loss is directly driven by depression-related characteristics. After passing through the classifier, the depression label corresponding to the short-term segment can be predicted. Therefore, the depression-related convolutional branch is used to actively retain discriminative information related to the depressive state, such as abnormal voice tone and negative facial expressions, and other multimodal cues related to depression detection, while suppressing information irrelevant to the classification task. In this embodiment, the classifier is a linear classifier used to map the 32-dimensional depression-related representation to a binary classification prediction output.

[0053] The depression-independent convolutional branch is not directly driven by the classification loss, but is mainly driven by the joint constraints of representation separation and reconstruction. The representation separation constraint is used to reduce the correlation between depression-related representations and depression-independent representations, making them as unrelated and non-overlapping as possible. The reconstruction constraint is used to ensure that the combination of depression-related and depression-independent representations can reconstruct the fragment-level fusion features of the input, thereby ensuring that the two decoupled branches still retain the effective information in the input features.

[0054] By minimizing the representation separation constraint, the correlation between the output representations of the two branches can be reduced, so that the depression-related convolution branch focuses on expressing information related to depression discrimination, while the depression-irrelevant convolution branch expresses information that is relatively unrelated to depression discrimination but helps to reconstruct the input features.

[0055] IV. Graph Attention Networks; By arranging the depression-related characteristics corresponding to all short segments within the same long video in chronological order, a long-term evidence sequence is constructed; and using each short segment as a graph node, a graph structure containing local temporal edges and global connectivity edges is built. It serves as both a short-time segment index and a graph node index.

[0056] Establish a long-term evidence sequence Represented as: ; in, For the first A short-term segment of depression-related characteristics.

[0057] In the Local Graph Attention (Local GAT) branch, edges are established between two nodes that satisfy the adjacency matrix in time, constructing local temporal edges. The specific adjacency matrix is ​​as follows: ; in, In constructing local temporal edges, the first The node and the first The adjacency matrix of nodes in time For the local time neighborhood, its value range is: Preferred .

[0058] In the Global Graph Attention (GAT) branch, an edge is established between any two nodes to construct a global connection edge, whose adjacency matrix can be represented as: ; in, In constructing global connected edges, the first The node and the first The adjacency matrix of nodes over time.

[0059] Next, the graph structure containing local temporal edges and global connection edges is input into the graph attention network GAT to model the information propagation relationship between nodes, obtain the updated representation of each node, and further form a global representation of the long video.

[0060] like Figure 3 As shown, the graph attention network uses short time segments as graph nodes, and each node... Indicates the first Each short-time segment corresponds to a node representation, whose initial value is the depression-related representation output by the representation decoupling module. The connections between nodes represent the association edges between short-time segments, used to describe the temporal proximity, skip connections, or global associations between different segments. Figure 3 In Represents a node For the target node Attention weights are used to measure the attention of nodes during updates. During representation, nodes The importance of the information carried. Based on the target node. For example, graph attention networks rely on their connected neighbor nodes. Calculate the corresponding attention coefficient The neighbor node information is then weighted and aggregated according to the normalized attention coefficient to obtain the updated node representation. Solid and dashed lines are used to indicate different types of node connections, and arrows indicate that information is propagated from neighboring nodes to the target node.

[0061] Specifically, firstly, the node features are input into the graph attention layer, and the nodes... With nodes Attention coefficient between The calculation is as follows: ; in, They are nodes and nodes The input feature vector (in the graph structure, each node corresponds to a short-term depression-related representation). It is a learnable linear transformation matrix used to map node features to a higher-dimensional or specific-dimensional attention space, thereby enhancing expressive power; This is a vector concatenation operation that concatenates the transformed features of two nodes along the dimensional direction into a single vector. This is a learnable attention weight vector with the same dimension as the concatenated vector, used to calculate the attention score (scalar) between two nodes. It is a non-linear activation function that allows small negative gradients and avoids completely suppressing negative inputs.

[0062] Attention coefficient After normalization, we get: ; in, Represents a node The set of adjacent nodes, For adjacent node indexes, The normalized attention coefficient represents the attention level of a node when aggregating neighbor information. The weight of the features; For nodes With nodes Attention coefficient between them.

[0063] Therefore, node The update result is: ; in, This represents a non-linear activation function, introducing a non-linear transformation into the graph attention network to enhance its expressive power. For nodes The node representation updated by the graph attention layer incorporates information about itself and its neighbors, and the weights are determined by the normalized attention coefficients. The decision is used for subsequent global aggregation.

[0064] After updating the representations of each node, all nodes in the graph structure are aggregated to obtain a global representation of the long video: ; in, This represents the graph-level readout function. For nodes The node representation after being updated by the graph attention layer. The global representation of the long video is obtained by performing global average pooling (GAP) (i.e., graph-level readout function) on all updated node representations. It has a fixed dimension and is used for final classification.

[0065] Input the global representation of the long video into the classifier, and output the depression detection results of the long video samples. Its output process can be represented as: ; in, For classifiers, This is a long-form video-based depression test result. It should be noted that... To characterize the first in the decoupling module Fragment-level intermediate prediction results of short time segments are used to assist in training depression-related representations; The final depression detection result is obtained by updating the representations of all short-term node segments and performing graph-level aggregation using a graph attention network.

[0066] In this embodiment, the training process of the depression detection model is as follows: This embodiment conducts experiments on the D-vlog dataset, which is divided into training, validation, and test sets. The input audio features are 25-dimensional, and the video features are 136-dimensional. After time alignment, missing frame imputation, and standardization of the two modal features, short time segments are segmented using a sliding window with a window length of 30 and a stride of 15. The D-Vlog dataset is a multimodal dataset for depression detection, consisting of YouTube videos, primarily used to develop depression detection models based on nonverbal behavior.

[0067] The training of the depression detection model is divided into three stages. The first stage trains the multimodal fusion module, in which the audio encoder and video encoder both use Transformer encoders with a hidden dimension of 256, a coding layer of 2, and an attention head of 4; the cross-modal attention embedding has a dimension of 64 and an attention head of 2.

[0068] In the second stage of training, the representation decoupling module uses a three-layer one-dimensional convolutional structure for both depression-related and depression-independent convolutional branches, with output channels of 512, 128, and 32 respectively. The total loss consists of classification loss, branch separation loss, and reconstruction loss.

[0069] The third stage trains the graph attention network, using 32-dimensional depression-related representations as graph node features; the temporal graph sets the local adjacency range to 5 and the jump connection step size to 10, while the fully connected graph establishes a connection between any two nodes.

[0070] The Adam optimizer was used in all stages of the experiment, with a random seed of 42, and weighted cross-entropy was employed to address class imbalance. During training, the best model was saved based on the F1 score on the validation set. Finally, accuracy, precision, recall, and F1 score were calculated on the test set as the experimental results. The final results achieved were 78.57% accuracy, 77.55% precision, 87.36% recall, and 82.16% F1 score, effectively improving the accuracy of the results.

[0071] Example; Visualization and training process of depression detection model, as shown below Figures 4-8 As shown.

[0072] t-SNE is a dimensionality reduction technique commonly used for visualizing high-dimensional data. It maps similar data points in a high-dimensional space to a low-dimensional space, making similar samples as close as possible in the low-dimensional space, while separating dissimilar samples as much as possible. t-SNE helps to intuitively observe the distribution of data, especially for complex classification tasks, as it can show the relative positions of samples from different classes in the feature space.

[0073] Figure 4 The results show the dimensionality reduction of features in a depression detection model using t-SNE. Figure 4 Each point in the diagram represents a sample; blue points represent normal samples, and orange points represent depressed samples. t-SNE dimensionality reduction reveals a clear separation between normal and depressed samples in two-dimensional space, indicating that the model successfully distinguishes these two types of samples during feature extraction.

[0074] also, Figure 4 The star-shaped markers indicate the center positions of the two types of samples, with blue stars representing the center of normal samples and orange stars representing the center of depressed samples. Figure 2 As can be observed, there is a relatively obvious gap between the centers of the two types of samples, which further illustrates that the present method can learn discriminative depressive characteristics in the feature extraction stage, thus providing effective support for subsequent classification tasks.

[0075] Figure 5 This example further illustrates the predicted probability distribution of normal and depressed samples. The horizontal axis represents the probability of a sample being predicted as depressed, and the vertical axis represents the corresponding density distribution. The blue bars represent normal samples, the orange bars represent depressed samples, and the red dashed line represents the classification decision boundary.

[0076] from Figure 5 As can be seen, the predicted probabilities of normal samples are mainly concentrated in the lower range, while the predicted probabilities of depressed samples are mainly concentrated in the higher range, indicating that the model can effectively distinguish between the two types of samples. Although there is still some overlap near the decision boundary, the overall distribution difference is quite significant, further validating the effectiveness of this method in the depression detection task.

[0077] from Figure 6As can be seen from the accuracy curves, the training set accuracy generally shows a continuous upward trend with increasing training epochs, gradually stabilizing in the later stages. This indicates that the model can continuously learn effective discriminative features from the training samples. While the validation set accuracy fluctuates somewhat, it also generally shows an upward trend and stabilizes at a high level in the later stages of training, indicating that the model still has good classification ability on data not used in training. The training accuracy being higher than the validation accuracy is normal, indicating that the model fits the training samples more fully; at the same time, the validation accuracy did not show a continuous decline, indicating that the model did not suffer from severe overfitting.

[0078] from Figure 7 The F1 score curve on the validation set shows that the F1 score fluctuates significantly in the early stages of training, indicating that the model's ability to distinguish between depressed and non-depressed samples is unstable in the initial phase. As the number of training epochs increases, the F1 score on the validation set gradually increases and stabilizes, remaining at a high level in the later stages. This demonstrates that the model not only improves the overall classification accuracy but also effectively balances precision and recall. Since depression detection tasks often suffer from class imbalance, the F1 score more comprehensively reflects the model's ability to identify the depression category; this curve shows that the proposed method effectively improves the discrimination performance for depression-related samples. from Figure 8 The loss curves show that the training set loss continuously decreases with increasing training epochs, indicating that the model parameter optimization process is effective and the training objective function guides the model to converge gradually. The validation set loss also shows an overall downward trend. Although there are some fluctuations in the middle, it gradually decreases and stabilizes in the later stages, indicating that the model's generalization error on the validation set gradually decreases. The training loss being lower than the validation loss is consistent with general training patterns, and the two do not show significant inverse divergence, indicating that the model can maintain good generalization ability while improving its training fitting ability.

[0079] Based on the above description of the embodiments, those skilled in the art will understand that the depression detection method, system, and medium based on long- and short-time multimodal temporal series modeling described in this embodiment can be implemented in pure software or deployed and run on a general-purpose or dedicated computing hardware platform. Based on this essence, the technical solution of this embodiment can be specifically implemented in the form of a software product containing program instructions. This software product can be stored on various non-volatile storage media or directly deployed as a local or cloud service. The program instructions are used to cause computer devices with processing capabilities—including but not limited to personal computers, server clusters, mobile terminals, or other network devices—to execute the steps described in this embodiment.

[0080] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A depression detection method of long-short time multi-modal time series modeling, characterized in that, include: The audio and video features of the video to be detected are obtained and segmented into multiple short segments; The audio and video features of each short segment are encoded and then fused together through cross-modal attention to obtain a segment-level fused representation; The segment-level fusion representation is decoupled into depression-related representations and depression-unrelated representations by a representation decoupling module, and trained based on classification loss, representation separation constraints and reconstruction constraints to extract depression-related representations for each short segment. Using each short video segment as a node and the depression-related representation of each short video segment as the initial representation of the node, a graph structure containing local temporal edges and global connection edges is constructed in chronological order. The node representations are updated and aggregated through a graph attention network. The obtained global representation of the long video is then classified, and the depression detection result is output.

2. The method of claim 1, wherein, The process of encoding and fusing audio and video features, followed by cross-modal attention, to obtain a segment-level fused representation is as follows: Audio features are encoded using an audio encoder to obtain audio modality-coded features; Video features are encoded using a video encoder to obtain video modality-coded features; Audio modality coding features and video modality coding features are each dimensionality-reduced using a convolutional neural network and then mapped to the same embedding space. In the embedding space, we construct the attention of audio to video and the attention of video to audio. We then perform residual fusion of the audio-to-video attention result with the dimensionality-reduced audio modal coding features and residual fusion of the video-to-audio attention result with the dimensionality-reduced video modal coding features. The two modal features after residual fusion are pooled separately and then concatenated to obtain a fragment-level fused representation.

3. The method of claim 1, wherein, The segment-level fusion representation is decoupled into depression-related and depression-independent representations through a representation decoupling module. Training is then performed based on classification loss, representation separation constraints, and reconstruction constraints to extract depression-related representations for each short-term segment. Specifically: The fragment-level fusion representation is input into a parallel set depression-related convolutional branch and a depression-independent convolutional branch to obtain depression-related representations and depression-independent representations. The separation and decoupling between depression-related representations and depression-independent representations are constrained by representation separation constraints. A fully connected classifier is used to classify and predict depression-related features in each short time segment, and the fully connected classifier and depression-related convolutional branch are optimized based on the classification loss. The classification loss, representation separation constraint, and reconstruction constraint are combined to form the optimization objective function, which is used to update the representation decoupling module.

4. The method of claim 1, wherein, Optimization objective function As follows: in, For classifying losses, To characterize the separation constraints, To divide and reconstruct constraints, and Indicates the weighting coefficient. The total number of short segments. For indexing short time segments, A true label of depression for a short period of time. For the first The prediction results of the classifier output from the depression-related characteristics of a short time segment. These are symptoms associated with depression. These are symptoms unrelated to depression. For the fragment-level fusion feature, the first One element, Represents the reconstruction of the first One element, This indicates the total number of characteristic elements.

5. The method according to claim 1, characterized in that, The construction of a graph structure containing local temporal edges and global connectivity edges in chronological order is specifically as follows: An edge will be established between two nodes that satisfy the adjacency matrix in time, thus constructing a local temporal edge; Connect any two nodes with an edge to construct a fully connected edge.

6. The method of claim 5, wherein, The adjacency matrix is ​​specifically as follows: ； wherein, is the adjacency matrix of the first node and the second node in time, is the local time neighborhood range.

7. The method of claim 1, wherein, The global representation of the long video is obtained by pooling all updated node representations.

8. The method of claim 1, wherein, In the classification of the long video global representation, the classification uses a fully connected layer to output the depression probability or category.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-8.

10. A computer readable storage medium characterized by, The computer-readable storage medium stores a plurality of computer programs, which are used to be invoked by a processor and to execute the method as described in any one of claims 1-8.

Images