Depression risk prediction method and system based on multi-modal data and computer readable storage medium

Abstract

The application relates to the technical field of deep learning, in particular to a depression risk prediction method and system based on multi-modal data and a computer readable storage medium. The method comprises the following steps: acquiring visual, audio and text data of a target object in a clinical interview, and performing feature extraction, segmentation and alignment based on a speech level time boundary; obtaining each modal context representation by using an independent context encoder; obtaining a speech level multi-modal fusion representation by cross-modal interaction through a symmetric cross fusion module; constructing a relationship time sequence diagram based on the fusion representation, and learning cross-speech structure dependency by using a graph neural network; and inputting the graph structure enhanced representation and the multi-modal fusion representation into a predictor after splicing, and outputting a depression severity prediction score. The method aims to solve the problems of insufficient modeling of speech level context dependency, cross-modal complementary information and cross-speech structure dependency in the prior art.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method, system, and computer-readable storage medium for predicting depression risk based on multimodal data. Background Technology

[0002] Depression is a common mental disorder, and its screening and severity assessment typically rely on professional interviews and scales. While these methods have significant clinical value, the assessment process and interpretation of results can still be affected by factors such as subjective reporting bias, scale measurement characteristics, the respondent's expressive state, and cross-population differences.

[0003] Currently, multimodal depression risk detection methods typically employ the following technical approach: First, the original features of the visual, audio, and text modalities are extracted separately. Then, the features of each modality are fused across modalities through methods such as feature concatenation, weighted summation, or attention fusion. Next, the fused features are input into a sequence model for temporal modeling. Finally, a depression severity prediction score is output based on the temporal modeling results.

[0004] However, in the specific application scenario of clinical interviews, the above-mentioned solutions often fail to meet the prediction accuracy requirements of practical applications. Firstly, regarding discourse-level contextual dependency modeling, the emotional expression and language content of the target subjects in clinical interviews often span multiple discourse rounds, exhibiting long-range semantic resonance and emotional accumulation effects. Existing methods typically treat the features of each discourse segment as independent samples or only capture short-range temporal dependencies through sequence models, making it difficult to effectively utilize cross-discourse contextual continuity cues. Secondly, regarding cross-modal interaction methods, feature splicing and static weighted fusion ignore the dynamic changes in intermodal interaction relationships, while single-modal guided attention fusion only grants one modality the authority to actively query information from other modalities, limiting the effectiveness of bidirectional complementary mechanisms between modalities. Thirdly, regarding temporal dependency modeling, sequential modeling can only propagate information step-by-step along the time axis, making it difficult to explicitly express forward backtracking, backward referencing, and jump-like temporal relationships between non-adjacent discourses. The three factors mentioned above are coupled with each other: context modeling requires full use of multimodal information, cross-modal fusion requires discourse-level context awareness, and temporal dependency modeling is based on high-quality fusion representation, making it difficult to achieve ideal results with improvements in a single dimension.

[0005] Therefore, how to achieve precise alignment of discourse-level multimodal information, full interaction of cross-modal information, and structured expression of cross-discourse temporal dependencies through a unified deep learning framework in clinical interview scenarios, thereby improving the accuracy of depression risk prediction, has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0006] The main purpose of this application is to provide a method for predicting depression risk based on multimodal data, which aims to solve the problem that existing methods are insufficient in modeling discourse-level contextual dependence, cross-modal complementary information, and cross-discourse structure dependence.

[0007] To achieve the above objectives, this application provides a method for predicting depression risk based on multimodal data, the method comprising:

[0008] S10, acquire multimodal data of the target object, wherein the multimodal data includes at least two of the following: visual modal data, audio modal data, and text modal data;

[0009] S20, based on the interview discourse-level time boundary, perform feature extraction, segmentation and time alignment on the multimodal data to obtain a multimodal feature sequence, and input the multimodal feature sequence into the context encoder corresponding to each modality to obtain a multimodal context representation;

[0010] S30, the multimodal context representation is subjected to intermodal interaction through a cross-modal attention mechanism to obtain a discourse-level multimodal fusion representation. A relational temporal graph is constructed based on the discourse-level multimodal fusion representation, wherein the nodes in the relational temporal graph represent discourse segments, and the edges represent the temporal dependencies between discourse segments.

[0011] S40, the relational time sequence graph is input into a graph neural network framework for graph representation learning to obtain an enhanced graph structure representation;

[0012] S50, the graph structure enhancement representation and the discourse-level multimodal fusion representation are jointly processed and input into the predictor to output the depression risk prediction score of the target object.

[0013] Optionally, the multimodal data includes video data, audio data, and text data transcribed from the audio data, and step S20 includes:

[0014] Visual features, acoustic features, and textual modal features are extracted from the multimodal data respectively;

[0015] The start and end times of each utterance segment are determined based on the text data, and the visual modal features and audio modal features are cropped based on the start and end times.

[0016] The cropped visual modal features, audio modal features, and corresponding text modal features are organized according to the discourse order to obtain a discourse-level aligned trimodal feature sequence.

[0017] Optionally, the context encoder includes a Transformer encoder and a fully connected layer, wherein:

[0018] The visual modality feature sequence, audio modality feature sequence, and text modality feature sequence are respectively input into the corresponding Transformer encoder for discourse-level context modeling, and the context representations of different modalities are mapped to a unified feature dimension through a fully connected layer;

[0019] For any mode Its discourse-level feature sequence is represented as:

[0020]

[0021] In the formula, , , These represent the visual modality, audio modality, and text modality, respectively. Indicates the number of speech fragments corresponding to the target object;

[0022] After processing by the context encoder, the context representation of this modality is obtained:

[0023]

[0024] In the formula, Representing modes The corresponding Transformer encoder, Representing modes The corresponding fully connected layer.

[0025] Optionally, the symmetric cross-fusion module includes multiple cross-modal attention paths, wherein:

[0026] For any two different modes and modal The context representation is used as the query vector, in modality. The context representation is used as a key vector and a value vector, which are computed through a cross-modal attention mechanism to obtain the modal... Information augmentation modality Cross-modal feature representation:

[0027]

[0028] The cross-modal attention mechanism is expressed as follows:

[0029]

[0030] In the formula, Represents the query vector. Represents the key vector. Represents a value vector. Represents the vector dimension.

[0031] For each target modality, the context representation of that target modality is added to the cross-modal features from the other two modalities to obtain the enhanced representation:

[0032]

[0033] In the formula, and To remove The other two modes;

[0034] The obtained visual enhancement representation Audio enhancement representation and text enhancement representation By concatenating the data, the utterance-level multimodal fusion representation is obtained:

[0035]

[0036] In the formula, This indicates that the pieces are joined along the feature dimension.

[0037] Optionally, the relational time sequence diagram is a directed graph. , including node set Edge set and set of relation types ,in:

[0038] Each node in the node set corresponds to a multimodal fusion representation of a speech fragment;

[0039] The edge set is used to represent the temporal dependencies between different discourse segments;

[0040] The set of relation types includes at least forward relations and backward relations, wherein the forward relation represents a temporal dependency from an earlier utterance to a later utterance, and the backward relation represents a temporal dependency from a later utterance to an earlier utterance;

[0041] Set the preset time window as For the first The first discourse node and the first A discourse node, when it satisfies and At the node With nodes Establish directed edges between them, and according to and The time sequence indicates the relationship type of the edge;

[0042] The edge weights are calculated using an attention mechanism or a learnable similarity function.

[0043] Optionally, the graph neural network framework includes a relational graph convolutional network, a graph Transformer network, and a GraphSAGE network connected in sequence, and S40 includes:

[0044] The relational graph convolutional network updates the node representations for different relation types in the relational time sequence graph;

[0045] The contribution of neighboring nodes to the current node's representation update is determined through the attention mechanism of the graph Transformer network.

[0046] The graph structure enhancement representation is obtained by aggregating the local neighborhood information of nodes through the GraphSAGE network.

[0047]

[0048] In the formula, Representing a sequence diagram of relationships, This represents the processing of relational graph convolutional networks. This represents graph Transformer network processing. This indicates GraphSAGE network processing.

[0049] Optionally, the predictor includes at least two fully connected layers and sets an activation function and a Dropout layer, wherein S50 includes:

[0050] The graph structure-enhanced representation is concatenated with the discourse-level multimodal fusion representation to obtain a joint representation corresponding to each discourse segment;

[0051] The predictor outputs a corresponding discourse-level depression severity prediction score based on the joint representation;

[0052] The predicted depression severity score for the target object is obtained by averaging all discourse-level depression severity prediction scores.

[0053]

[0054] In the formula, This indicates the number of interview snippets from the target interviewee. Indicates the first The predicted score for the severity of depression corresponding to each discourse segment. The score indicates the predictive score of the target individual's depression severity.

[0055] Optionally, during model training, a joint loss function is used to optimize the prediction results. This joint loss function includes consistency correlation coefficient loss, mean squared error loss, and a standard deviation consistency constraint term, expressed as:

[0056]

[0057] In the formula, The correlation coefficient represents consistency. This represents the mean square error. The standard deviation of the predictive score for depression severity is represented by the following: The standard deviation of the true severity score of depression;

[0058] The consistency correlation coefficient is expressed as:

[0059]

[0060] In the formula, This represents the Pearson correlation coefficient between predicted depression severity scores and actual depression severity scores. This represents the mean score predicting the severity of depression. This represents the mean score of the true severity of depression.

[0061] The mean square error is expressed as:

[0062]

[0063] In the formula, This represents the true value of the depression severity score. The predicted value of the depression severity score.

[0064] Furthermore, to achieve the above objectives, this application also provides a depression risk prediction system based on multimodal data, the system comprising:

[0065] The data acquisition module is used to acquire visual modal data, audio modal data, and text modal data of the target subject in a clinical interview scenario;

[0066] The feature processing module is used to extract, segment, and time-align features from the visual modal data, audio modal data, and text modal data based on discourse-level temporal boundaries.

[0067] The context encoding module is used to perform context modeling on visual modal feature sequences, audio modal feature sequences, and text modal feature sequences respectively, to obtain the corresponding context representations;

[0068] The symmetric cross-fusion module is used to perform intermodal interaction through a cross-modal attention mechanism to obtain a discourse-level multimodal fusion representation;

[0069] The graph construction module is used to construct a relational sequence graph based on the discourse-level multimodal fusion representation;

[0070] The graph learning module is used to learn graph representations of the relational time series graph through a graph neural network framework to obtain an enhanced graph structure representation.

[0071] The prediction module is used to output a predicted score of the target object's depression severity based on the graph structure augmentation representation and the discourse-level multimodal fusion representation.

[0072] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the depression risk prediction method based on multimodal data as described above.

[0073] This application has at least the following beneficial effects:

[0074] 1. This application segments and aligns visual modal data, audio modal data, and text modal data in time based on the time boundary of the interview discourse, so that each discourse segment contains multimodal information corresponding to its semantic boundary, which helps to reduce the impact of mismatch of different modalities in the time dimension on subsequent modeling.

[0075] 2. This application models visual, audio and text features through an independent context encoder and a symmetric cross-fusion module, which enhances the complementary information interaction between modalities while preserving the expressive characteristics of each modality.

[0076] 3. This application constructs a relational time sequence graph based on discourse-level multimodal fusion representation and learns cross-discourse structure dependencies through a graph neural network framework. At the same time, the graph structure enhancement representation and discourse-level multimodal fusion representation are concatenated for prediction, enabling the model to utilize both modality fusion information and interview structure information simultaneously, providing an achievable technical solution for the auxiliary assessment of multimodal depression severity in clinical interview scenarios. Attached Figure Description

[0077] Figure 1 This is a flowchart illustrating the depression risk prediction method based on multimodal data involved in the embodiments of this application;

[0078] Figure 2 This is a schematic diagram of the overall structure of the multimodal depression risk detection model involved in the embodiments of this application;

[0079] Figure 3 This is a schematic diagram of the symmetrical cross-fusion module structure involved in the embodiments of this application;

[0080] Figure 4 This is a schematic diagram of the relationship sequence graph construction and graph learning structure involved in the embodiments of this application;

[0081] Figure 5This is a schematic diagram of a depression risk prediction system based on multimodal data involved in an embodiment of this application;

[0082] Figure 6 This is a schematic diagram of the hardware operating environment of the depression risk prediction system based on multimodal data involved in the embodiments of this application;

[0083] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0084] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0085] Example 1

[0086] Reference Figure 1 This embodiment provides a method for predicting depression risk based on multimodal data, the method comprising the following steps:

[0087] S10, acquire multimodal data of the target object, wherein the multimodal data includes at least two of the following: visual modal data, audio modal data, and text modal data;

[0088] In clinical interview scenarios, the target subject (i.e. the person being assessed) and the interviewer engage in semi-structured or structured question-and-answer dialogues, a process that simultaneously generates three different physical sources of information.

[0089] The multimodal data referred to in this application refers to information from three different modalities: visual modal data, audio modal data, and text modal data collected during the same clinical interview. Visual modal data includes nonverbal behavioral video signals of the target subject, such as facial expressions, head posture, and body movements, recorded by a camera device. Audio modal data includes speech signals collected by a sound pickup device, which contains paralinguistic information such as prosodic features, energy variations, spectral characteristics, and vocal stability. Text modal data includes semantic text content obtained from the speech signals through automatic speech recognition (ASR) tools or manual transcription. These three modalities reflect the target subject's emotional state and behavioral performance from different physical dimensions, each carrying some complementary behavioral cues—for example, the target subject may deny depressive emotions through verbal expression (text modal), but facial expressions (visual modal) or changes in tone of voice (audio modal) may reveal their true emotional state. Therefore, the necessity of joint analysis of the three modalities constitutes the starting point of this embodiment.

[0090] S20, based on the interview discourse-level time boundary, perform feature extraction, segmentation and time alignment on the multimodal data to obtain a multimodal feature sequence, and input the multimodal feature sequence into the context encoder corresponding to each modality to obtain a multimodal context representation;

[0091] Specifically, transcribed texts are obtained from interview audio, and the start and end times of each discourse segment in the transcribed text are determined. Acoustic features are extracted from the audio modality data, visual features are extracted from the visual modality data, and semantic encoding is performed on the text modality data. The audio modality features and visual modality features are cropped according to the time boundaries of each discourse segment, and the cropped audio modality features, visual modality features, and corresponding text modality features are organized in discourse order.

[0092] In this embodiment, since depression assessment relies on understanding the semantic content of discourse, segmenting a statement into two fixed windows would disrupt its semantic integrity in each window, thus affecting the accuracy of subsequent semantic analysis. Therefore, using discourse-level temporal boundaries as anchor points for multimodal feature alignment, continuous multimodal signals are segmented into semantically complete discourse units, ensuring that the granularity of feature alignment remains consistent with the semantic boundaries, rather than using the fixed-time-window slicing method commonly used in existing technologies.

[0093] It is worth noting that, unlike the traditional use of fixed time windows such as 2 to 5 seconds, which may not only destroy semantic integrity but also lead to unavoidable mismatches at the positions where different modalities are cut, this step keeps the granularity of the segmentation consistent with the semantic boundary of the discourse. One modality is aligned exactly with the signal transition area at the window boundary, while another modality contains a complete semantic unit within the window. The discourse-level time boundary is used as an anchor point to avoid mismatches.

[0094] In one alternative implementation, audio modal features include prosody, energy, spectrum, or vocal stability features; visual modal features include facial expressions, head posture, facial action units, eye movement information, or depth visual representations; and text modal features are obtained by encoding transcribed text using a pre-trained language model.

[0095] Thus, three discourse-level feature sequences are obtained:

[0096]

[0097]

[0098]

[0099] In the formula, , , These represent the visual modality feature sequence, the audio modality feature sequence, and the text modality feature sequence, respectively. This indicates the number of interview snippets corresponding to the target subject. , , They represent the first The visual modal features, audio modal features, and text modal features corresponding to each discourse fragment.

[0100] Specifically, the context encoder includes a Transformer encoder and a fully connected layer, with no parameter sharing between the context encoders corresponding to different modalities. The Transformer encoder is used to perform context modeling on the modal feature sequence arranged in utterance order, and the fully connected layer is used to map the context representations of different modalities to a unified feature dimension. For any modality ,in Its context encoding process can be represented as:

[0101]

[0102] In the formula, Representing modes The discourse-level feature sequence, Representing modes The corresponding Transformer encoder, Representing modes The corresponding fully connected layer, Representing modes The context representation.

[0103] Through the above processing, a visual context representation is obtained. Audio context representation and text context representation .

[0104] In this embodiment, in a clinical interview scenario, the target subject's emotional expression often spans multiple discourse rounds. For example, a depressed patient may exhibit emotional blunting in the early stages of the interview, but gradually reveal low mood in the later stages. Analyzing a single discourse segment alone cannot perceive this cross-discourse emotional change trajectory. Therefore, the context encoding in step S20 is designed to address this issue. Furthermore, the obtained context representations of the three modalities are processed through a cross-modal attention mechanism so that each modality can obtain supplementary information from the other two modalities.

[0105] S30, the multimodal context representation is subjected to intermodal interaction through a cross-modal attention mechanism to obtain a discourse-level multimodal fusion representation. A relational temporal graph is constructed based on the discourse-level multimodal fusion representation, wherein the nodes in the relational temporal graph represent discourse segments, and the edges represent the temporal dependencies between discourse segments.

[0106] Specifically, refer to Figure 3 For any two different modes and ,in and modal The context representation is used as the query vector, in modality. The context representation is used as the key vector and value vector, and the result is obtained from the modality. Information augmentation modality Cross-modal feature representation:

[0107]

[0108] in:

[0109]

[0110] For the target mode Its context representation is compared with that from the other two modalities. and The cross-modal features are added together to obtain the enhanced representation:

[0111]

[0112] Through the above processing, a visual enhancement representation is obtained. Audio enhancement representation and text enhancement representation .

[0113] Finally, the enhanced representations of the three modalities are concatenated to obtain the discourse-level multimodal fusion representation:

[0114]

[0115] in, This indicates that the pieces are joined along the feature dimension.

[0116] Through the aforementioned symmetrical cross-fusion process, each modality can be used as a query modality to obtain complementary information from other modalities.

[0117] It should be noted that, as a common phenomenon in clinical interviews, patients retrospectively mention details of early experiences in later utterances, or emotional foreshadowing in early utterances gradually unfolds in subsequent utterances. Traditional techniques, such as LSTM or Transformer, which use unidirectional stepwise modeling along the time axis, struggle to directly trace back to the original information of specific utterances multiple time steps ago. Therefore, this step also uses a relational sequence graph to transform temporal dependencies from implicit state propagation to an explicit graph topology representation. This ensures a direct connection path between any two utterance nodes satisfying window constraints, unaffected by the asymptotic limitation of sequence length, thus enabling explicit modeling of leaping semantic associations between non-adjacent utterances.

[0118] Specifically, the relational sequence diagram is a directed graph, which can be represented as:

[0119]

[0120] In the formula, Represents a set of nodes. Denotes the set of edges. This represents a set of relation types. Each node in the node set corresponds to a multimodal fusion representation of a utterance fragment, and the edge set represents the temporal dependencies between different utterance fragments. The relation type set includes at least forward relations. and backward relation .

[0121] Specifically, for the first Each utterance fragment can have its node representation set as the corresponding multimodal fusion representation. When constructing the relational sequence diagram, not only are temporal connections established between adjacent discourse nodes, but also skip-temporal connections are established between non-adjacent discourse nodes within a preset time window. Let the preset time window be... ,in This indicates the size of the discourse window used for backtracking. This indicates the size of the discourse window for subsequent connections. For the... The first discourse node and the first A discourse node, when the following condition is met:

[0122]

[0123] At the node With nodes Establish directed edges between them. When When this happens, the directed edge is marked as a forward relation. ;when When this happens, the directed edge is marked as a backward relation. .

[0124] In one alternative implementation, the edge weights can be represented as:

[0125]

[0126] In the formula, and They represent the first The and the first Feature representation of each discourse node. Representing relation type The corresponding learnable weight matrix, Represents a node With nodes In relation The edge weights below.

[0127] S40, the relational time sequence graph is input into a graph neural network framework for graph representation learning to obtain an enhanced graph structure representation;

[0128] Reference Figure 4 The graph neural network framework comprises a relational graph convolutional network, a graph Transformer network, and a GraphSAGE network connected in sequence. The relational graph convolutional network updates the representation of discourse nodes according to different relation types in the relational temporal graph, modeling the impact of different temporal directions such as forward and backward relations on node representations. The graph Transformer network determines the contribution of neighboring nodes to the current node representation update based on an attention mechanism, enhancing cross-discourse dependency modeling capabilities. The GraphSAGE network aggregates local neighborhood information of nodes to obtain an enhanced graph structure representation.

[0129] In one alternative implementation, the discourse-level multimodal fusion representation is... After inputting the graph neural network framework, the graph structure augmentation representation can be expressed as:

[0130]

[0131] In the formula, This represents the time sequence diagram of the relationship. This represents the processing of relational graph convolutional networks. This represents graph Transformer network processing. This indicates GraphSAGE network processing.

[0132] Specifically, the node update process of the relational graph convolutional network can be represented as follows:

[0133]

[0134] In the formula, Representing relations Next node The set of neighboring node indexes, and Representing relations The edge weight below, and This represents the learnable weight matrix. This is the discourse-level multimodal fusion representation obtained through the symmetric cross-fusion module.

[0135] Subsequently, the node representations output by the relational graph convolutional network are used as input to the Transformer network. For each node... and its neighboring nodes The Graph Transformer network can calculate the contribution weights of neighboring nodes to the current node through an attention mechanism:

[0136]

[0137] And update the node representation according to the contribution weight:

[0138]

[0139] In the formula, The graph Transformer network represents the first... Each self-attention point (total) indivual), , , and Both represent learnable weight matrices. This represents the hidden layer dimension of each self-attention head.

[0140] Furthermore, the node representations output by the Graph Transformer network are input into the GraphSAGE network to aggregate the local neighborhood information of the nodes, thus obtaining the node representations. Graph structure enhancement representation:

[0141]

[0142] In the formula, , For a trainable weight matrix, Represents a node The set of neighboring nodes.

[0143] After graph learning, we obtain the graph structure-enhanced representations corresponding to each utterance fragment of the target object:

[0144]

[0145] In the formula, This indicates the number of speech fragments corresponding to the target object.

[0146] S50, the graph structure enhancement representation and the discourse-level multimodal fusion representation are jointly processed and input into the predictor to output the depression risk prediction score of the target object.

[0147] Furthermore, for the first A discourse fragment, its graphical structure is enhanced representation. Representation of discourse-level multimodal fusion By concatenating the components, we obtain the joint representation:

[0148]

[0149] The joint representation is input into the predictor to obtain a speech-level depression severity prediction score:

[0150]

[0151] In the formula, Indicates the first The predicted score for the severity of depression corresponding to each discourse segment.

[0152] The predicted depression severity score for the target individual is obtained by averaging all discourse-level depression severity prediction scores.

[0153]

[0154] In the formula, This indicates the number of speech fragments corresponding to the target object. The score indicates the predictive score of the target individual's depression severity.

[0155] In the technical solution provided in this embodiment, trimodal alignment is achieved through discourse-level temporal boundaries, contextual information of each modality is modeled through an independent context encoder, complementary information interaction between vision, audio and text is enhanced through a symmetric cross-fusion module, and cross-discourse structural dependencies are modeled through a relational temporal graph and graph neural network framework, thereby providing an achievable technical solution for the auxiliary assessment of depression severity in clinical interview scenarios.

[0156] Example 2

[0157] As an optional embodiment, this embodiment provides a method for constructing and training a multimodal depression risk detection model based on the first embodiment.

[0158] Reference Figure 2 The multimodal depression risk detection model includes a feature alignment module, a context encoding module, a symmetric cross-fusion module, a graph construction module, a graph learning module, and a prediction module.

[0159] During training, the trimodal discourse-level feature sequence of the target object is used as the model input, and the depression severity score corresponding to the target object is used as the supervision label. The depression severity score can be a PHQ-8 score, a PHQ-9 score, or other continuous scores used to represent the severity of depression.

[0160] In each training round, the model sequentially performs feature alignment, context encoding, symmetric cross-fusion, relational temporal graph construction, graph representation learning, and prediction to obtain a predicted depression severity score. Subsequently, a joint loss function is calculated based on the predicted score and the actual depression severity score, and the learnable parameters in the model are updated based on backpropagation until a preset number of training rounds is reached or a preset convergence condition is met.

[0161] In an alternative implementation, the joint loss function includes consistency correlation coefficient loss, mean squared error loss, and standard deviation consistency constraint term, which can be expressed as:

[0162]

[0163] In the formula, The correlation coefficient represents consistency. This represents the mean square error. The standard deviation of the predictive score for depression severity is represented by the following: The standard deviation of the true severity score of depression.

[0164] The consistency correlation coefficient is expressed as:

[0165]

[0166] In the formula, This represents the Pearson correlation coefficient between predicted depression severity scores and actual depression severity scores. This represents the mean of the predicted scores. This represents the mean of the true scores.

[0167] The mean square error is expressed as:

[0168]

[0169] In the formula, Indicates the number of training samples. Indicates the first Predicted depression severity scores for each training sample. Indicates the first The true depression severity scores of each training sample.

[0170] The above training method enables the model to optimize the numerical error of prediction while taking into account the consistency between the prediction results and the true labels, and to constrain the difference in standard deviation between the predicted distribution and the true distribution.

[0171] Example 3

[0172] This embodiment further explains the implementation conditions and experimental results of the multimodal depression risk detection model proposed in the previous embodiments.

[0173] (1) Simulation experimental conditions

[0174] In one specific implementation, the experimental code was written in Python 3.11, based on the PyTorch 2.7.0 deep learning framework and the PyTorch Geometric 2.7.0 graph neural network tool, and the model was trained and tested on a computing device configured with an Intel Core i7-12700KF CPU, an NVIDIA GeForce RTX 3090Ti GPU, and sufficient memory.

[0175] In one alternative implementation, the uniform hidden dimension in the model can be set to 200, and the forward and backward time windows in the relational sequence graph construction can be set to... The attention heads of the graph Transformer network were set to 4, the hidden dimension of the first fully connected layer of the prediction module was set to 100, and the dropout rate was set to 0.5. During training, the Adam optimizer was used for model tuning, with the learning rate set to 0.00025, the batch size set to 50, and the number of training epochs set to 500.

[0176] It should be noted that the above parameters are only one possible implementation. In other implementations, the hidden dimension, time window size, number of attention heads, number of training epochs, learning rate, and batch size can be adjusted according to the dataset size, input feature dimension, computing resources, and validation set performance.

[0177] (2) Simulation Experiment Content and Result Analysis

[0178] In a specific experiment, the multimodal depression risk detection model was validated on the publicly available interview dataset E-DAIC. The method of this application was compared with existing multimodal depression risk detection methods through simulation to obtain the PHQ-8 depression severity prediction score, and the consistency correlation coefficient (CCC) and root mean square error (RMSE) were used as evaluation indicators. CCC measures the consistency between the predicted score and the actual score; a higher CCC indicates better consistency. RMSE measures the error between the predicted score and the actual score; a lower RMSE indicates a smaller prediction error. The simulation results are shown in Table 1.

[0179] surface The prediction results of this application and the prior art in the simulation experiment

[0180]

[0181] As shown in Table 1, under the simulation experimental conditions, the proposed method achieves a CCC score of 0.634, higher than the comparative methods in the table, indicating good consistency between its predicted and actual scores. Compared to DepressNet, the proposed method improves the CCC from 0.457 to 0.634 and reduces the RMSE from 5.36 to 4.85, demonstrating that under the experimental conditions, discourse-level context encoding, symmetric cross-fusion, and relational time-series graph learning help utilize multimodal interview information. Compared to TensorFormer, the proposed method achieves a higher CCC, but its RMSE does not reach the same level, indicating that the proposed method performs better in consistency modeling, while numerical error control can still be further optimized.

[0182] The simulation results above demonstrate that, under the experimental conditions, the depression risk prediction method based on multimodal data proposed in this application can predict depression severity scores and achieves good simulation results in terms of consistency correlation coefficient. It should be noted that the above simulation results are only used to illustrate the feasibility and effectiveness of the technical solution under specific experimental conditions and do not constitute a limitation on the scope of protection, applicable datasets, feature extraction methods, or model parameter settings of this application.

[0183] Example 4

[0184] As one implementation, this application also provides a depression risk prediction system based on multimodal data. (See reference...) Figure 5 The system may include a data acquisition module, a feature processing module, a context encoding module, a symmetric cross-fusion module, a graph construction module, a graph learning module, and a prediction module.

[0185] The data acquisition module is used to acquire visual modal data, audio modal data, and text modal data of the target subject in a clinical interview scenario; the feature processing module includes a feature extraction module and a feature alignment module. The feature extraction module is used to extract features from each modal data, and the feature alignment module performs segmentation and time alignment based on discourse-level temporal boundaries; the context encoding module is used to obtain visual context representation, audio context representation, and text context representation; the symmetric cross-fusion module is used to obtain discourse-level multimodal fusion representation; the graph construction module is used to construct a relational temporal graph; the graph learning module is used to obtain graph structure enhancement representation; and the prediction module is used to output a predicted score for the depression severity of the target subject.

[0186] As one implementation scheme, Figure 6 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0187] like Figure 6 As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0188] Those skilled in the art will understand that Figure 6 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0189] like Figure 6 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0190] exist Figure 6In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0191] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:

[0192] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0193] S10, acquire multimodal data of the target object, wherein the multimodal data includes at least two of the following: visual modal data, audio modal data, and text modal data;

[0194] S20, based on the interview discourse-level time boundary, perform feature extraction, segmentation and time alignment on the multimodal data to obtain a multimodal feature sequence, and input the multimodal feature sequence into the context encoder corresponding to each modality to obtain a multimodal context representation;

[0195] S30, the multimodal context representation is subjected to intermodal interaction through a cross-modal attention mechanism to obtain a discourse-level multimodal fusion representation. A relational temporal graph is constructed based on the discourse-level multimodal fusion representation, wherein the nodes in the relational temporal graph represent discourse segments, and the edges represent the temporal dependencies between discourse segments.

[0196] S40, the relational time sequence graph is input into a graph neural network framework for graph representation learning to obtain an enhanced graph structure representation;

[0197] S50, the graph structure enhancement representation and the discourse-level multimodal fusion representation are jointly processed and input into the predictor to output the depression risk prediction score of the target object.

[0198] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0199] Visual features, acoustic features, and textual modal features are extracted from the multimodal data respectively;

[0200] The start and end times of each utterance segment are determined based on the text data, and the visual modal features and audio modal features are cropped based on the start and end times.

[0201] The cropped visual modal features, audio modal features, and corresponding text modal features are organized according to the discourse order to obtain a discourse-level aligned trimodal feature sequence.

[0202] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0203] The visual modality feature sequence, audio modality feature sequence, and text modality feature sequence are respectively input into the corresponding Transformer encoder for discourse-level context modeling, and the context representations of different modalities are mapped to a unified feature dimension through a fully connected layer;

[0204] For any mode Its discourse-level feature sequence is represented as:

[0205]

[0206] In the formula, , , These represent the visual modality, audio modality, and text modality, respectively. Indicates the number of speech fragments corresponding to the target object;

[0207] After processing by the context encoder, the context representation of this modality is obtained:

[0208]

[0209] In the formula, Representing modes The corresponding Transformer encoder, Representing modes The corresponding fully connected layer.

[0210] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0211] For any two different modes and modal The context representation is used as the query vector, in modality. The context representation is used as a key vector and a value vector, which are computed through a cross-modal attention mechanism to obtain the modal... Information augmentation modality Cross-modal feature representation:

[0212]

[0213] The cross-modal attention mechanism is expressed as follows:

[0214]

[0215] In the formula, Represents the query vector. Represents the key vector. Represents a value vector. Represents the vector dimension.

[0216] For each target modality, the context representation of that target modality is added to the cross-modal features from the other two modalities to obtain the enhanced representation:

[0217]

[0218] In the formula, and To remove The other two modes;

[0219] The obtained visual enhancement representation Audio enhancement representation and text enhancement representation By concatenating the data, the utterance-level multimodal fusion representation is obtained:

[0220]

[0221] In the formula, This indicates that the pieces are joined along the feature dimension.

[0222] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0223] A relational time sequence graph is constructed based on the aforementioned discourse-level multimodal fusion representation. The relational time sequence graph is a directed graph. , including node set Edge set and set of relation types ,in:

[0224] Each node in the node set corresponds to a multimodal fusion representation of a speech fragment;

[0225] The edge set is used to represent the temporal dependencies between different discourse segments;

[0226] The set of relation types includes at least forward relations and backward relations, wherein the forward relation represents a temporal dependency from an earlier utterance to a later utterance, and the backward relation represents a temporal dependency from a later utterance to an earlier utterance;

[0227] Set the preset time window as For the first The first discourse node and the first A discourse node, when it satisfies and At the node With nodes Establish directed edges between them, and according to and The time sequence indicates the relationship type of the edge;

[0228] The edge weights are calculated using an attention mechanism or a learnable similarity function.

[0229] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0230] The relational graph convolutional network updates the node representations for different relation types in the relational time sequence graph;

[0231] The contribution of neighboring nodes to the current node's representation update is determined through the attention mechanism of the graph Transformer network.

[0232] The graph structure enhancement representation is obtained by aggregating the local neighborhood information of nodes through the GraphSAGE network.

[0233]

[0234] In the formula, Representing a sequence diagram of relationships, This represents the processing of relational graph convolutional networks. This represents graph Transformer network processing. This indicates GraphSAGE network processing.

[0235] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0236] The graph structure-enhanced representation is concatenated with the discourse-level multimodal fusion representation to obtain a joint representation corresponding to each discourse segment;

[0237] The predictor outputs a corresponding discourse-level depression severity prediction score based on the joint representation;

[0238] The predicted depression severity score for the target object is obtained by averaging all discourse-level depression severity prediction scores.

[0239]

[0240] In the formula, This indicates the number of interview snippets from the target interviewee. Indicates the first The predicted score for the severity of depression corresponding to each discourse segment. The score indicates the predictive score of the target individual's depression severity.

[0241] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0242] During model training, a joint loss function is used to optimize the prediction results. This joint loss function includes consistency correlation coefficient loss, mean squared error loss, and standard deviation consistency constraint term, expressed as:

[0243]

[0244] In the formula, The correlation coefficient represents consistency. This represents the mean square error. The standard deviation of the predictive score for depression severity is represented by the following: The standard deviation of the true severity score of depression;

[0245] The consistency correlation coefficient is expressed as:

[0246]

[0247] In the formula, This represents the Pearson correlation coefficient between predicted depression severity scores and actual depression severity scores. This represents the mean score predicting the severity of depression. This represents the mean score of the true severity of depression.

[0248] The mean square error is expressed as:

[0249]

[0250] In the formula, This represents the true value of the depression severity score. The predicted value of the depression severity score.

[0251] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0252] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the depression risk prediction method based on multimodal data as described in the above embodiments.

[0253] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0254] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0255] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0256] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0257] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0258] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0259] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0260] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for predicting depression risk based on multimodal data, characterized in that, The depression risk prediction method based on multimodal data includes: S10, acquire multimodal data of the target object, wherein the multimodal data includes at least two of the following: visual modal data, audio modal data, and text modal data; S20, based on the interview discourse-level time boundary, perform feature extraction, segmentation and time alignment on the multimodal data to obtain a multimodal feature sequence, and input the multimodal feature sequence into the context encoder corresponding to each modality to obtain a multimodal context representation; S30, the multimodal context representation is subjected to intermodal interaction through a cross-modal attention mechanism to obtain a discourse-level multimodal fusion representation. A relational temporal graph is constructed based on the discourse-level multimodal fusion representation, wherein the nodes in the relational temporal graph represent discourse segments, and the edges represent the temporal dependencies between discourse segments. S40, the relational time sequence graph is input into a graph neural network framework for graph representation learning to obtain an enhanced graph structure representation; S50, the graph structure enhancement representation and the discourse-level multimodal fusion representation are jointly processed and input into the predictor to output the depression risk prediction score of the target object.

2. The depression risk prediction method based on multimodal data as described in claim 1, characterized in that, The multimodal data includes video data, audio data, and text data transcribed from the audio data. S20 includes: Visual features, acoustic features, and textual modal features are extracted from the multimodal data respectively; The start and end times of each utterance segment are determined based on the text data, and the visual modal features and audio modal features are cropped based on the start and end times. The cropped visual modal features, audio modal features, and corresponding text modal features are organized according to the discourse order to obtain a discourse-level aligned trimodal feature sequence.

3. The depression risk prediction method based on multimodal data as described in claim 1, characterized in that, The context encoder includes a Transformer encoder and a fully connected layer, wherein: The visual modal feature sequence, audio modal feature sequence, and text modal feature sequence are respectively input into the corresponding Transformer encoder for discourse-level context modeling, and the context representations of different modalities are mapped to a unified feature dimension through a fully connected layer; For any mode Its discourse-level feature sequence is represented as: ； In the formula, , , These represent the visual modality, audio modality, and text modality, respectively. Indicates the number of speech fragments corresponding to the target object; After processing by the context encoder, the context representation of this modality is obtained: ； In the formula, Representing modes The corresponding Transformer encoder, Representing modes The corresponding fully connected layer.

4. The depression risk prediction method based on multimodal data as described in claim 1, characterized in that, The symmetric cross-fusion module includes multiple cross-modal attention paths, wherein: For any two different modes and In modal The context representation is used as the query vector, in modality. The context representation is used as a key vector and a value vector, which are computed through a cross-modal attention mechanism to obtain the modal... Information augmentation modality Cross-modal feature representation: ； The cross-modal attention mechanism is expressed as follows: ； In the formula, Represents the query vector. Represents the key vector. Represents a value vector. Indicates the dimension of a vector; For each target modality, the context representation of that target modality is added to the cross-modal features from the other two modalities to obtain the enhanced representation: ； In the formula, and To remove The other two modes; The obtained visual enhancement representation Audio enhancement representation and text enhancement representation By concatenating the data, the utterance-level multimodal fusion representation is obtained: ； In the formula, This indicates that the pieces are joined along the feature dimension.

5. The depression risk prediction method based on multimodal data as described in claim 1, characterized in that, The relational sequence diagram is a directed graph. , including node set Edge set and set of relation types ,in: Each node in the node set corresponds to a multimodal fusion representation of a speech fragment; The edge set is used to represent the temporal dependencies between different discourse segments; The set of relation types includes at least forward relations and backward relations. The forward relation represents a temporal dependency from an earlier utterance to a later utterance, and the backward relation represents a temporal dependency from a later utterance to an earlier utterance. Set the preset time window as For the first The first discourse node and the first A discourse node, when it satisfies and At the node With nodes Establish directed edges between them, and according to and The time sequence indicates the relationship type of the edge; The edge weights are calculated using an attention mechanism or a learnable similarity function.

6. The depression risk prediction method based on multimodal data as described in claim 1, characterized in that, The graph neural network framework includes a relational graph convolutional network, a graph Transformer network, and a GraphSAGE network connected in sequence. S40 includes: The relational graph convolutional network updates the node representations for different relation types in the relational time sequence graph; The contribution of neighboring nodes to the current node's representation update is determined through the attention mechanism of the graph Transformer network. The graph structure enhancement representation is obtained by aggregating the local neighborhood information of nodes through the GraphSAGE network. ； In the formula, Representing a sequence diagram of relationships, This represents the processing of relational graphs using convolutional networks. This represents graph Transformer network processing. This indicates GraphSAGE network processing.

7. The depression risk prediction method based on multimodal data as described in claim 1, characterized in that, The predictor includes at least two fully connected layers and sets an activation function and a Dropout layer. S50 includes: The graph structure-enhanced representation is concatenated with the discourse-level multimodal fusion representation to obtain a joint representation corresponding to each discourse segment; The predictor outputs a corresponding discourse-level depression severity prediction score based on the joint representation; The predicted depression severity score for the target object is obtained by averaging all discourse-level depression severity prediction scores. ； In the formula, This indicates the number of interview snippets from the target interviewee. Indicates the first The predicted score for the severity of depression corresponding to each discourse segment. The score indicates the predictive score of the target individual's depression severity.

8. The depression risk prediction method based on multimodal data as described in claim 1, characterized in that, During model training, a joint loss function is used to optimize the prediction results. This joint loss function includes consistency correlation coefficient loss, mean squared error loss, and standard deviation consistency constraint term, expressed as: ； In the formula, The correlation coefficient represents consistency. This represents the mean square error. The standard deviation of the predictive score for depression severity is represented by the following: The standard deviation of the true severity score of depression; The consistency correlation coefficient is expressed as: ； In the formula, This represents the Pearson correlation coefficient between predicted depression severity scores and actual depression severity scores. This represents the mean score predicting the severity of depression. The mean of the true severity score of depression; The mean square error is expressed as: ； In the formula, This represents the true value of the depression severity score. The predicted value of the depression severity score.

9. A depression risk prediction system based on multimodal data, characterized in that, The system includes: The data acquisition module is used to acquire visual modal data, audio modal data, and text modal data of the target subject in a clinical interview scenario; The feature processing module is used to extract, segment, and time-align features from the visual modal data, audio modal data, and text modal data based on discourse-level temporal boundaries. The context encoding module is used to perform context modeling on visual modal feature sequences, audio modal feature sequences, and text modal feature sequences respectively, to obtain the corresponding context representations; The symmetric cross-fusion module is used to perform intermodal interaction through a cross-modal attention mechanism to obtain a discourse-level multimodal fusion representation; The graph construction module is used to construct a relational sequence graph based on the discourse-level multimodal fusion representation; The graph learning module is used to learn graph representations of the relational time series graph through a graph neural network framework to obtain an enhanced graph structure representation. The prediction module is used to output a predicted score of the target object's depression severity based on the graph structure augmentation representation and the discourse-level multimodal fusion representation.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the depression risk prediction method based on multimodal data as described in any one of claims 1 to 8.

Images