A hierarchical expert collaboration-based multi-modal depression state detection method and system and a storage medium

Abstract

The application provides a multi-modal depression state detection method and system based on hierarchical expert collaboration, and a storage medium, the method comprising the following steps: step 1: defining core depression factors from a causal perspective, including social ability factors, mental state factors, and medication history factors; step 2: using a speech codec FACodec framework to explicitly separate the input original audio; step 3: using a large language model to generate semantic time stamps when summarizing core depression factors from interview text, and extracting aligned frame-level features from the corresponding non-verbal signals based on the semantic time stamps; step 4: constructing a double-stream expert structure of parallel text expert streams and audio-video expert streams; and step 5: outputting the final evaluation result through dynamic modal gating. The application has the following beneficial effects: 1. The discrimination accuracy of the model in a complex environment is significantly improved; and 2. A large amount of unstructured text is converted into structured clinical evidence, and non-causal noise is filtered.

Description

Technical Field

[0001] This invention relates to the field of medical and health information processing technology, and in particular to a method, system and storage medium for detecting multimodal depressive states based on hierarchical expert collaboration. Background Technology

[0002] Depression is a complex mental health disorder clinically characterized by persistent low mood, loss of interest, and anhedonia. Current diagnosis heavily relies on subjective assessments by psychiatrists and standardized self-assessment scales (such as the PHQ-9 or HAMD). However, this traditional diagnostic model has three major limitations: First, high subjective bias—due to the lack of objective physiological quantitative indicators, different doctors may interpret the same patient differently, leading to low diagnostic consistency; second, uneven resource distribution—a shortage of professional mental health resources prevents many patients from receiving timely early screening, increasing behavioral risks; and third, limited information dimensions—traditional scales primarily capture patients' explicit verbal statements, easily overlooking nonverbal cues revealed during interviews (such as flat tone of voice and avoidant eye contact), which often more accurately reflect subconscious states.

[0003] To address these issues, researchers began exploring AI-based automated depression detection technologies. Early techniques focused on psycholinguistic features, such as using the LIWC tool to count the frequency of first-person pronouns or words associated with negative emotions. Subsequently, Large Language Models (LLMs) based on the Transformer architecture were introduced, utilizing self-attention mechanisms to capture long-range semantic dependencies. To compensate for the limitations of single text, existing technologies attempt to integrate the patient's language content (Text), acoustic prosody (Audio), and visual behavior (Video). In current academia and industry, automated methods for predicting depressive states can be broadly categorized into the following typical technologies: Modality-aware fusion models: Representative models include TFN (Tensor Fusion Network), MulT (Multimodal Transformer), and MISA. These models focus on the mathematical integration of cross-modal features. For example, TFN captures intermodal interactions by calculating the Cartesian product of features; MISA attempts to project representations into modality-specific and modality-invariant subspaces to enhance the robustness of fusion.

[0004] Context-aware sequence models: Representative models include bc-LSTM (bidirectional contextual LSTM), Speechformer, and GRU / BiLSTM-based methods. Due to the long temporal characteristics of clinical interviews, these models utilize recurrent neural networks or hierarchical Transformer architectures to capture long-range dependencies and sequence patterns in the dialogue.

[0005] Topic- and hierarchical models: Representative models include Topic Modeling and HiQuE (Hierarchical Question Embedding Network). Models such as HiQuE attempt to distinguish between "main questions" and "follow-up questions" in interviews, improving diagnostic performance by explicitly modeling the dialogue structure.

[0006] Direct diagnosis using large language models: Due to the rapid development of large language models in recent years, some methods utilize the powerful semantic understanding capabilities of pre-trained models to directly perform zero-sample or few-sample depression discrimination on interview transcripts.

[0007] Despite the progress made by the aforementioned baseline techniques on standard datasets, the following subtle yet profound shortcomings remain in practical clinical applications and in processing complex interview data: Models such as bc-LSTM and the traditional Transformer typically treat interviews as a flat, linear sequence. However, in real interviews, doctors often include a lot of meaningless content to make the conversation sound natural and to soothe the patient's emotions. These models cannot consciously filter out irrelevant details from massive amounts of dialogue like professional doctors can, resulting in their attention mechanisms failing to accurately focus on discriminative pathological features. Furthermore, when processing continuous signals like audio and video, they often employ global pooling or fixed window segmentation, which is not very effective at capturing transient pathological manifestations.

[0008] Modeling methods like HiQuE rely on a rigid division of the "question-answer" structure. In real clinical interviews, conversations are often unstructured and open-ended, and some datasets (such as E-DAIC-WOZ) even lack interviewer question records. Due to its over-reliance on "questions" as semantic anchors, HiQuE's hierarchical modeling ability significantly decreases when there is a lack of question context or a variable conversation structure, making it unable to independently capture key features from the interviewee's one-sided statements.

[0009] Using end-to-end large language models directly for judgment also presents similar problems to sequence modeling. Clinical interviews typically contain tens of thousands of lexical units, filled with a large amount of redundant details such as social pleasantries that are not related to pathology. Due to the low signal-to-noise ratio of long texts, general large models are prone to attention diffusion and induce "hallucinations," making it impossible to filter out dimensions that are meaningful for diagnosis like professional doctors. Furthermore, using only the text modality means losing non-linguistic information, resulting in low accuracy in diagnosis.

[0010] Meanwhile, most existing technologies are "black box" systems, employing fixed computational paths. Clinician diagnoses are based on comprehensive assessments of specific symptom clusters (such as sleep disorders and social withdrawal), and different patients exhibit different primary pathological modalities (some manifest in speech, others in facial expressions). Existing models cannot dynamically adjust their focus based on the patient's specific manifestations. For example, when a patient verbally conceals information, the model cannot spontaneously select more reliable audio-visual physiological cues for collaborative correction. Furthermore, most current methods directly fuse pre-trained audio features with sufficient semantic meaning with textual modalities. This approach results in severe coupling between features, blurring the boundaries of the prediction space and thus affecting the model's predictive ability. Researchers refer to this problem as "feature entanglement."

[0011] The information disclosed in this background section is intended only to enhance understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0012] To address the issues of long text sparsity, audio-video feature entanglement, and the tendency to overlook weak modalities in multimodal fusion during clinical interviews, this invention proposes a multimodal depressive state prediction method based on hierarchical expert collaboration.

[0013] This invention provides a multimodal depressive state detection method based on hierarchical expert collaboration, including... As a further improvement of the present invention, step 1: setting core depression factors from a causal perspective, the core depression factors include social competence factors, mental state factors, and medication history factors; wherein, the social competence factors are used to assess the patient's sources of life stress and interpersonal relationship management; the mental state factors are used to assess the patient's internal emotions, sleep quality, and changes in appetite circadian rhythm; the medication history factors are used to record the patient's past medical history, treatment interventions, and drug dosages. Step 2: Use the FACodec speech codec framework to explicitly separate the input raw audio to obtain the separated audio components; Step 3: When summarizing core depressive factors from interview texts using a large language model, generate semantic timestamps, and extract aligned frame-level features from the corresponding non-linguistic signals based on the semantic timestamps; Step 4: Construct a parallel dual-stream expert structure consisting of a text expert stream and an audio / video expert stream. The text expert stream is used to extract text features from the interview text and output text prediction results, while the audio / video expert stream is used to extract multimodal features from the audio / video signals and output audio / video prediction results. Step 5: Concatenate the global language representation vector output by the text expert stream with the non-language representation vector output by the audio and video expert stream, map the result through a multilayer perceptron to obtain a global context joint feature vector, and output the final evaluation result through dynamic modal gating.

[0014] As a further improvement of the present invention, step 2 further includes: Step 20: Using vector quantization and adversarial training mechanisms, the original continuous audio signal is decomposed into mutually independent attribute subspaces, including: content encoding subspace, prosodic encoding subspace, and timbre encoding subspace; wherein, the content encoding subspace is used to capture text semantics, the prosodic encoding subspace is used to represent the dynamic changes of pitch, energy, and speech rate, and the timbre encoding subspace is used to characterize identity features. Step 21: Remove the content encoding components of the content encoding subspace and retain the prosodic representation of the prosodic encoding subspace as the final input feature of the audio modality.

[0015] As a further improvement of the present invention, step 3 further includes: Step A, Semantic Timestamp Generation: Input the interview content into the large language model. When summarizing the core depressive factors, add constraint instructions through prompting engineering, requiring the model to explicitly output the original dialogue segment numbers and corresponding timestamp intervals on which the summary is based. ,in , t start It is the start time of the original dialogue paragraph corresponding to the summary section. t end It is the time when the dialogue segment ends. N The representative large model obtained N The summary content has the following corresponding points: N One original dialogue segment; Step B, alignment and extraction of non-linguistic features: based on the semantic timestamp Aligned frame-level features are extracted from the corresponding non-linguistic signals.

[0016] The present invention also discloses a multimodal depressive state detection system based on hierarchical expert collaboration, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the multimodal depressive state detection method of the present invention when called by the processor.

[0017] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the multimodal depressive state detection method of the present invention when invoked by a processor.

[0018] The beneficial effects of this invention are: 1. In the features extracted by general speech pre-training models, text semantics and acoustic prosody are highly coupled (feature entanglement), resulting in redundant audio-text modal information and obscuring the truly diagnostic acoustic emotional cues. To solve this problem, this invention introduces the FACodec speech encoding and decoding framework, which uses vector quantization and adversarial training to explicitly decompose audio into content, prosody, and timbre subspaces. By actively removing redundant content encodings, only pure prosodic representations are retained as input, ensuring that audio features can serve as an independent supplementary dimension to text semantics, and significantly improving the model's discrimination accuracy in complex environments. 2. Clinical interview texts are extremely long and full of social greetings and other noise. End-to-end models are prone to "long text forgetting" and induce semantic "illusions". To solve this problem, this invention uses the causal reasoning ability of a large language model to actively decouple the three core depressive factors of "social competence", "mental state" and "medication history" from the original dialogue, transforming massive amounts of unstructured text into structured clinical evidence and filtering out non-causal noise. 3. Traditional methods perform global pooling on the entire video, which can erase the instantaneous micro-expressions or intonation abnormalities that the subject exhibits at specific moments (such as when discussing trauma). This invention proposes a "semantic timestamp" alignment mechanism, which requires LLM to explicitly trace the corresponding dialogue segments and physical timestamps while summarizing factors. This achieves fine-grained alignment at the "semantic anchor" level. The system only extracts audiovisual features and performs sequence modeling within key time intervals, effectively filtering out background interference and preserving local high-value pathological evidence. Attached Figure Description

[0019] Figure 1 This is a block diagram illustrating the principle of the multimodal depressive state detection method of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0021] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0022] Parameter explanation: DSM-5-TR: Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition - Text Revised Edition); VGG-16 Network: Deep Convolutional Neural Network; The purpose of this invention is to overcome the shortcomings of existing depression detection technologies in processing long clinical interviews, such as low signal-to-noise ratio, feature entanglement, and lack of clinical logical support. It proposes a multimodal depressive state prediction method based on hierarchical expert collaboration. This invention aims to utilize the logical reasoning capabilities of a large language model to actively extract core factors directly causally related to depression from redundant dialogues. By transforming unstructured long texts into structured clinical evidence, it removes non-pathological noise such as social small talk at the source, improving the model's ability to focus on key pathological features. By introducing speech encoding and decoding decoupling technology, it explicitly decomposes the speech signal and removes the semantic content of the text to solve the problems of semantic overlap and obscuring emotional cues. This invention utilizes semantic evidence points traced by a large model to locate precise timestamp intervals, thereby extracting highly synchronized audiovisual features and preserving the subject's instantaneous micro-expressions and acoustic prosodic features under specific topics. Addressing the problem that existing "black box" systems lack reasoning processes and are difficult to deploy in medical systems, this invention aims to simulate the logic of clinicians' "multi-dimensional comprehensive assessment" through a hybrid expert architecture.

[0023] like Figure 1 As shown, this invention achieves dynamic coordination and correction of prediction bias for a single text modality by explicitly decoupling audio signals, establishing a semantic-level fine-grained timestamp mapping, and designing a hierarchical expert system and a diversity gating mechanism. The specific technical solution is as follows: Step 1: To overcome non-pathological noise interference such as small talk and pauses in long texts, this invention first defines three core depressive factors from a causal perspective. Core Factor Definition: Based on DSM-5-TR guidelines, three dimensions with causal relationships are clearly defined: Social competence (SA): Focus on the sources of stress in the patient's life and their ability to handle interpersonal relationships.

[0024] Mental state (MS): Examines changes in internal emotions, sleep quality, and physiological rhythms such as appetite.

[0025] Medication history (MH): Pay attention to past medical history, treatment interventions, and drug dosage.

[0026] Step 2: Explicit Attribute Decoupling Scheme for Speech Features To address the issue of highly coupled (feature entanglement) linguistic and prosodic content in features generated by pre-trained speech models, this invention introduces the FACodec framework for explicit audio separation. Utilizing Vector Quantization (VQ) and adversarial training mechanisms, the original continuous audio signal is decomposed into three independent attribute subspaces: content encoding, prosodic encoding, and timbre and acoustic details. Content encoding captures textual semantics, timbre encoding characterizes identity features, and prosodic encoding focuses on representing the dynamic changes in pitch, energy, and speech rate. This invention proactively removes redundant content encoding components, retaining only pure prosodic representations as the final input features for the audio modality. This design fundamentally alleviates the semantic conflict between speech and text modalities, ensuring that acoustic features can serve as an independent, complementary dimension, capturing depressive indicators such as slow speech rate, low voice, or subtle tremors.

[0027] Step 3: Fine-grained feature alignment mechanism based on semantic timestamps To avoid diluting transient pathological evidence with traditional coarse-grained alignment, this invention proposes an alignment mechanism based on "semantic timestamps." We input interview content into a large language model. When using the large language model to summarize core depressive factors (social skills, mental state, medication history) from the interview text, we add constraint instructions through prompting engineering, requiring the model to explicitly output the original dialogue segment numbers and corresponding timestamp intervals upon which the summary is based. ,in , t start It is the start time of the original dialogue paragraph corresponding to the summary section. t end It is the time when the dialogue segment ends. N The representative large model obtained N The summary content has the following corresponding points: N The first original dialogue segment. For non-linguistic feature extraction, a VGG-16 network is first used to extract frame-level features. For the first... semantic timestamp intervals Average pooling is performed over time to obtain fragment-level visual representations. , It is the set of real numbers. The number of dimensions representing visual features. Within the same interval. Extracting the decoupled pure prosodic features from the inner segment, and then using pooling to obtain the fragment-level acoustic representation. , It is the set of real numbers. The number of dimensions representing audio features cascades the visual and acoustic representations of each segment along the dimensions into a joint representation. The non-verbal features were arranged in chronological order of the physical time of the interviews. The data is then input into a Transformer encoder for global context modeling. Global average pooling is performed on the hidden state sequence of the Transformer to obtain the final global multimodal non-linguistic representation vector. Details can be seen as follows Figure 1 As shown.

[0028] Step 4: Layered Multimodal Hybrid Expert System Architecture This invention designs a parallel two-stream expert structure to achieve multi-level feature mining: Text Hybrid Expert Flow: LLM decouples social competence (SA), mental state (MS), and medication history (MH) from the original interview and maps them to a high-dimensional dense feature vector through encoding and pooling layers. , and For each combination of depressive factors in the text (single factor, two factors, and all factors), we train a corresponding expert network, forming seven multi-expert networks for the three depressive factor combinations. This module also includes a module for evaluating the utility of the depressive factors, selecting the most suitable expert for evaluation based on the information from the three depressive factors. Finally, the stream outputs independent prediction results for the text modality. .

[0029] Audio / Video Expert Stream: Receives global multimodal non-linguistic representation vectors Pre-train an audio / video expert to predict non-verbal signal states, based on Output the independent predicted probability distribution of the subject's state .

[0030] Step 5: Collaborative Correction and Anti-Collapse Mechanism The audiovisual expert flow is not simply a feature aggregation, but rather serves as a "correction factor" for semantic modality. We concatenate the three factor feature vectors of the text along their feature dimensions to obtain the global language representation vector. , (Social ability) is a text feature corresponding to social competence. It is mental state and Medication history, and its correlation with non-verbal representation vectors. After concatenation, the global context joint feature vector is obtained by mapping through a multilayer perceptron. The transformed features, i.e., the global context joint feature vector from step 51. Input Dynamic Modality Gate (DMG): To achieve exclusive routing that selects the output of a specific modality as the final result, multimodal gating also introduces the Gumbel-Softmax reparameterization technique. This technique can reparameterize continuous fractions while maintaining the differentiability of the computational graph. Mapped to discrete two-dimensional one-hot routing vectors , The weights representing the text modalities. This represents the weights of the audio and video modalities. These weights are one-hot (only 0 / 1), which differs from the scores below. Specifically, to achieve discrete routing, we first output continuous scores to the gating network. Noise from a standard Gumbel distribution is introduced to simulate a random sampling process. For modal branching... The activation value with noise is calculated as follows: in, It is random noise sampled independently and identically distributed from the standard Gumbel distribution, and the calculation formula is: ,in , u It is a real number randomly sampled from the interval (0, 1). This is the temperature coefficient. When... At this time, the distribution approaches a standard one-hot distribution, thus achieving hard route selection; however, during training, a larger... To maintain output smoothness, the value should be greater than or equal to 1, typically between 1 and 2, ensuring proper gradient backpropagation. During the inference phase, we employ a Straight-Through estimator. The final discrete routing vector is obtained. , It is the probability value calculated in the gumble-softmax formula above.

[0031] The system's final diagnostic output The modality is entirely determined by the modality expert selected by the gating network. .

[0032] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multimodal depressive state detection method based on hierarchical expert collaboration, characterized in that, include: Step 1: Define core depression factors from a causal perspective. These core depression factors include social competence factors, mental state factors, and medication history factors. The social competence factors are used to assess the patient's sources of stress and interpersonal relationship management. The mental state factors are used to assess the patient's internal emotions, sleep quality, and changes in appetite circadian rhythms. The medication history factors are used to record the patient's past medical history, treatment interventions, and medication dosages. Step 2: Use the FACodec speech codec framework to explicitly separate the input raw audio to obtain the separated audio components; Step 3: When summarizing core depressive factors from interview texts using a large language model, generate semantic timestamps, and extract aligned frame-level features from the corresponding non-linguistic signals based on the semantic timestamps; Step 4: Construct a parallel dual-stream expert structure consisting of a text expert stream and an audio / video expert stream. The text expert stream is used to extract text features from the interview text and output text prediction results, while the audio / video expert stream is used to extract multimodal features from the audio / video signals and output audio / video prediction results. Step 5: Concatenate the global language representation vector output by the text expert stream with the non-language representation vector output by the audio and video expert stream, map the result through a multilayer perceptron to obtain a global context joint feature vector, and output the final evaluation result through dynamic modal gating.

2. The multimodal depressive state detection method according to claim 1, characterized in that, Step 2 also includes: Step 20: Using vector quantization and adversarial training mechanisms, the original continuous audio signal is decomposed into mutually independent attribute subspaces, including: content encoding subspace, prosody encoding subspace, and timbre encoding subspace; wherein, the content encoding subspace is used to capture text semantics, the prosody encoding subspace is used to represent the dynamic changes of pitch, energy, and speech rate, and the timbre encoding subspace is used to characterize identity features. Step 21: Remove the content encoding components of the content encoding subspace and retain the prosodic representation of the prosodic encoding subspace as the final input feature of the audio modality.

3. The multimodal depressive state detection method according to claim 1, characterized in that, Step 3 also includes: Step A, Semantic Timestamp Generation: Input the interview content into the large language model. When summarizing the core depressive factors, add constraint instructions through prompting engineering, requiring the model to explicitly output the original dialogue segment numbers and corresponding timestamp intervals on which the summary is based. ,in , t start It is the start time of the original dialogue paragraph corresponding to the summary section. t end It is the time when the dialogue segment ends. N The representative large model obtained N The summary content has the following corresponding points: N One original dialogue segment; Step B, alignment and extraction of non-linguistic features: based on the semantic timestamp Aligned frame-level features are extracted from the corresponding non-linguistic signals.

4. The multimodal depressive state detection method according to claim 3, characterized in that, Step B further includes: Step B1: Extract frame-level features using the VGG-16 network; Step B2: For the first semantic timestamp intervals Average pooling is performed over time to obtain fragment-level visual representations. , It is the set of real numbers. The number of dimensions representing visual features; within the same semantic timestamp interval Extracting the decoupled pure prosodic features from the inner segment, and then using pooling to obtain the fragment-level acoustic representation. , It is the set of real numbers. The number of dimensions representing audio features cascades the visual and acoustic representations of each segment along the dimensions to form a joint representation. The non-verbal features were arranged in chronological order of the physical time of the interviews. ; Step B3: Input the Transformer encoder to perform global context modeling; Step B4: Perform global average pooling on the hidden state sequence of the Transformer to obtain the final global multimodal non-linguistic representation vector. .

5. The multimodal depressive state detection method according to claim 1, characterized in that, Step 4 also includes: Step a, Text Hybrid Expert Flow: Using a large language model, three core depressive factors—social competence, mental state, and medication history—are decoupled from the original interview text and encoded and pooled into high-dimensional dense feature vectors. These are the characteristics of text corresponding to social skills. It is mental state and The study included medication history; an expert network was trained for each non-empty combination of the core depressive factors, resulting in seven expert networks; based on the types of depressive factors activated in the input text, the corresponding expert network was selected for evaluation, and independent prediction results for the text modality were output. ; Step b, Audio / Video Expert Flow: Receive global multimodal non-linguistic representation vectors. Pre-trained audio-visual experts that independently predict non-verbal signal states, based on Output the independent predicted probability distribution of the subject's state .

6. The multimodal depressive state detection method according to claim 1, characterized in that, Step 5 also includes: Step 50: Concatenate the three factor feature vectors of the text along the feature dimension to obtain the global language representation vector. ,in Features of text that represent social skills Indicates mental state, Indicate medication history; Step 51: Convert the global language representation vector from step 50... With multimodal non-linguistic representation vectors After concatenation, the global context joint feature vector is obtained by mapping through a multilayer perceptron. ; Step 52: Global Context Joint Feature Vector from Step 51 Input dynamic modal gating DMG: M It represents a continuous fraction.

7. The multimodal depressive state detection method according to claim 6, characterized in that, In step 52, the dynamic modal gating DMG introduces the Gumbel-Softmax reparameterization technique to convert continuous fractions while maintaining the differentiability of the computation graph. Mapped to discrete two-dimensional one-hot routing vectors , The weights representing the text modalities. This represents the weights of the audio and video modalities.

8. The multimodal depressive state detection method according to claim 7, characterized in that, Step 52 also includes: Step a1: Output continuous fractions to the gating network Noise from a standard Gumbel distribution is introduced to simulate a random sampling process. The weight score represents the text modality. The weighted score represents the audio / video modality. Step a2: For modal branching Its activation value contains noise. The calculation is as follows: in, It is random noise sampled independently and identically distributed from the standard Gumbel distribution, and the calculation formula is: ,in , u It is a real number randomly sampled from the interval (0, 1); For temperature coefficient; when At this time, the distribution approaches the standard one-hot distribution, thus achieving hard route selection; during training, values ​​greater than or equal to 1 It can maintain the smoothness of the output and ensure that the gradient can be backpropagated normally; during the inference phase, a straight-through estimator is used. The final discrete routing vector is obtained. , It is a probability value; Step a3: Final diagnostic output The modality expert selected by the gating network decides: Indicates independent prediction results. This represents the probability distribution of independent predictions.

9. A multimodal depressive state detection system based on hierarchical expert collaboration, characterized in that, include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the multimodal depressive state detection method according to any one of claims 1-8 when invoked by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the multimodal depressive state detection method according to any one of claims 1-8 when invoked by a processor.

Images