A multi-modal depression recognition system based on large language model evidence reasoning

Abstract

The present application relates to the field of depression state auxiliary identification, and discloses a multi-modal depression identification system based on large language model evidence reasoning, comprising a multi-modal data processing module, a feature encoding module, an evidence reasoning module, an evidence guided fusion module and a classification explanation module. The present application is used to solve the problems in the prior art that the local short-time emotion and stable depression related features are lack of hierarchical division in multi-modal depression state identification, and the semantic explanation generated by the large language model is separated from the multi-modal feature fusion and classification process, realize the extraction, correlation and hierarchical aggregation of low-intensity emotional clues scattered in different text contexts, reduce the interference of emotional ambiguity on depression state identification, and improve the traceability and explainability of the identification process.

Description

Technical Field

[0001] This invention relates to the field of auxiliary identification of depressive states, and more specifically to a multimodal depression identification system based on evidence reasoning using a large language model. Background Technology

[0002] With the development of artificial intelligence and multimodal information processing technologies, automatic depression identification technology has gradually evolved from single text, voice, or video analysis to multimodal identification methods that combine text semantics, speech acoustics, and visual behavioral information. Existing methods typically extract semantic and emotional features from interview texts, pitch, energy, speech rate, and pause features from audio, and facial movements, gaze, and head posture features from videos. These features are then fused through feature concatenation, attention mechanisms, graph neural networks, or multimodal Transformers to obtain a depression category or severity score.

[0003] Existing technologies still have at least two shortcomings: First, there is a lack of hierarchical distinction between instantaneous, situational negative emotions and stable depression-related traits that recur across contexts, which can easily lead to the over-amplification of isolated emotional cues or the neglect of scattered weak cues; Second, the explanations generated by some large language models are separated from the multimodal feature fusion and classification processes, making it difficult to explain how the original text fragments, stable features, and multimodal behaviors work together to influence the recognition results, resulting in insufficient traceability and reliability of the recognition process. Summary of the Invention

[0004] This invention provides a multimodal depression recognition system based on evidence reasoning using a large language model. It addresses the problems in existing multimodal depression state recognition, such as the lack of hierarchical distinction between local short-term emotions and stable depression-related features, and the separation between semantic interpretation generated by the large language model and the fusion and classification processes of multimodal features. The system extracts, associates, and hierarchically aggregates low-intensity emotional cues scattered across different text contexts, reducing the interference of emotional ambiguity on depression state recognition and improving the traceability and interpretability of the recognition process.

[0005] This invention is achieved through the following technical solution:

[0006] A multimodal depression recognition system based on evidence reasoning using a large language model includes:

[0007] Multimodal data processing module: used to acquire multimodal interaction data of the target object, extract features and perform time correspondence on the multimodal interaction data to obtain a first feature; the multimodal interaction data includes text data, audio data and video data;

[0008] Feature encoding module: used to encode the first feature to obtain the second feature;

[0009] Evidence reasoning module: used to construct a hierarchical evidence chain based on the text data in the multimodal interaction data through a large language model, enhance the second feature and perform cross-modal feature interaction based on the hierarchical evidence chain to obtain the enhanced second feature; also used to encode the evidence reasoning items in the hierarchical evidence chain to obtain the third feature;

[0010] Evidence-guided fusion module: used to perform feature aggregation on the enhanced second feature, determine the evidence-guided fusion weight, and obtain the fourth feature based on the evidence-guided fusion weight and the third feature;

[0011] The classification and interpretation module is used to input the fourth feature into the depression state classification model, obtain the classification result, and generate an interpretable report.

[0012] To address the shortcomings of existing multimodal depression identification technologies, such as the lack of hierarchical distinction between local short-term mood and stable depression-related features, and the separation between semantic interpretation generated by large language models and the multimodal feature fusion and classification processes, this invention proposes a multimodal depression identification system based on evidence reasoning using a large language model. In this system: a multimodal data processing module extracts features and performs time-correlation analysis on text, audio, and video data; a feature encoding module forms unified-dimensional modal encoding features; an evidence reasoning module constructs hierarchical evidence chains, which are then used for fine-grained feature enhancement and high-level evidence representation to obtain enhanced second and third features; an evidence-guided fusion module completes multimodal fusion and obtains a fourth feature; and a classification and interpretation module outputs the depression state classification results and an interpretability report based on the evidence chains.

[0013] This application is able to extract, associate, and hierarchically aggregate low-intensity emotional cues scattered in different textual contexts, reducing the interference of emotional ambiguity on the identification of depressive states, and significantly improving the traceability and interpretability of the identification process.

[0014] Furthermore, the multimodal data processing module includes:

[0015] Text processing unit: used to perform sentence segmentation, word segmentation, and linguistic information extraction on the text data to obtain first text features;

[0016] Audio processing unit: used to resample, segment, denoise, and extract acoustic features from the audio data to obtain the first audio feature;

[0017] Video processing unit: used to perform face detection, facial key point localization, and visual behavior feature extraction on the video data to obtain the first video features.

[0018] Time correspondence unit: used to correspond the first text feature, the first audio feature, and the first video feature to the same interview segment or time window based on the timestamps of the text data, audio data, and video data.

[0019] In this scheme, the multimodal data processing module extracts features from text, audio, and video data generated during the same interview or question-and-answer process, and maps the three modalities to the same interview segment or time window based on timestamps to obtain the first text feature, the first audio feature, and the first video feature. A modal mask is established based on the availability of different modal data within each time window. The modal mask is used to filter out invalid cross-modal interaction paths and participates in subsequent fusion weight normalization.

[0020] The linguistic information includes at least one of word information, part-of-speech information, keyword information, emotion-related linguistic information, and cognitive-related linguistic information; the acoustic features include at least one of Mel frequency cepstral coefficients, pitch, energy, speech rate, pauses, and prosodic features; and the visual behavioral features include at least one of facial action units, gaze direction, head posture, and facial expression change features.

[0021] Furthermore, the feature encoding module includes:

[0022] Text encoder: used to encode the first text features;

[0023] Audio timing encoder: used to encode the first audio feature;

[0024] Video temporal encoder: used to encode the first video features;

[0025] Feature mapping unit: used to map the encoding results of text encoder, audio temporal encoder and video temporal encoder to a unified dimension, to obtain the second text feature, the second audio feature and the second video feature respectively, and combine them to obtain the second feature.

[0026] This scheme encodes the first text feature, the first audio feature, and the first video feature separately using a feature encoding module. Then, it maps the three encoding results to a unified feature dimension through feature mapping to obtain the second text feature, the second audio feature, and the second video feature. The encoding method can employ context encoding or temporal encoding, among others.

[0027] Furthermore, the evidence reasoning module constructs a hierarchical chain of evidence using the following method:

[0028] The text data in the multimodal interaction data is divided into several local text contexts;

[0029] The local text context is input into the large language model, and multiple instantaneous emotion items are obtained based on the large language model to represent local short-term emotional states, resulting in a set of instantaneous emotion items;

[0030] The text data in the multimodal interaction data and the set of transient emotion items are input into a large language model. Based on the large language model, the recurrence, persistence and cross-context consistency of each transient emotion item in different local text contexts are analyzed to obtain multiple stable feature items used to characterize stable depression-related traits, and thus obtain a set of stable feature items.

[0031] The text data, instantaneous sentiment item set, and stable feature item set in the multimodal interaction data are input into the large language model. Based on the large language model, the evidence association relationship between each instantaneous sentiment item and each stable feature item is determined, and a directional evidence path from the instantaneous sentiment item to the stable feature item is established to obtain the evidence reasoning item.

[0032] The set of instantaneous emotion items, the set of stable feature items, and the set of evidence reasoning items are used as the hierarchical evidence chain.

[0033] In this scheme, the hierarchical evidence chain includes a set of transient emotion items, a set of stable feature items, and evidence reasoning items. For example, when corresponding to the i-th target object, the set of transient emotion items is denoted as S. i The set of stable feature terms is denoted as T. i The evidence reasoning item is denoted as E. i The hierarchical chain of evidence is denoted as follows: .

[0034] Local text context refers to text segments divided from the complete interview according to sentence boundaries, question-and-answer rounds, or topic changes, while retaining the corresponding sentence position, question-and-answer rounds, character position, and time information to limit the triggering range of local emotional states and establish the location relationship between evidence items and the original text.

[0035] Transient emotion items are used to characterize short-term emotional states triggered by specific events, topics, or local text contexts. Each transient emotion item includes at least the emotion category, emotion polarity, emotion intensity, triggering context, original text evidence fragment, location information, and transient emotion item confidence, enabling the local emotional state to be mapped to the original text location that formed that state. The j-th transient emotion item for the i-th target object can be denoted as... .

[0036] Stable features are used to characterize relatively stable depression-related features formed when multiple transient emotion items repeatedly appear, persist, and maintain semantic consistency in different local text contexts. The formation of stable features comprehensively considers the recurrence, persistence, and cross-contextual consistency of transient emotion items. Recurrence reflects the repeated occurrence of the relevant state in different contexts; persistence reflects the continuation of the relevant state within the interview timeframe; and cross-contextual consistency reflects whether the same or similar emotional, cognitive, or behavioral tendencies are presented under different events or topics. The k-th stable feature of the i-th target object can be denoted as... .

[0037] The evidence reasoning section is used to organize supporting evidence, counter-evidence, and uncertain evidence between transient emotion items and stable characteristic items, and records the correlation information between "emotion-cognition-behavior" and the transient emotion items. Pointing to stable feature terms The directional evidence path is defined as follows: supporting evidence indicates that the local transient state positively supports the corresponding stable feature; opposing evidence indicates that the relevant expression reflects information contrary to the target stable feature or retains interests or adaptive functions; uncertain evidence indicates that the local state has a clear event trigger, alternative explanation, conflicting evidence, or insufficient evidence strength. The directional evidence path is used to indicate the organization of evidence and the direction of feature transmission, and does not indicate a causal relationship in medicine.

[0038] As can be seen, the evidence reasoning module of this scheme constructs a set of transient emotion items, a set of stable feature items, and evidence reasoning items sequentially based on text data. The large language model first extracts transient emotion items from the local text context; then, it combines the complete text to analyze the recurrence, persistence, and cross-contextual consistency of relevant transient emotion items to form stable feature items; finally, it combines the complete text and the two-level evidence items to determine supporting, opposing, and uncertain evidence relationships, obtaining evidence reasoning items. The "two-level evidence" refers to two levels: transient emotion items and stable feature items. The first level of evidence is transient emotion items, used to characterize short-term emotional states triggered by specific events, topics, or local text contexts; the second level of evidence is stable feature items, used to characterize relatively stable features formed when multiple transient emotion items repeatedly appear, persist, and maintain a certain semantic consistency in different local text contexts.

[0039] This hierarchical generation process is used to distinguish between event-induced local emotions and depression-related features that are stable across contexts, rather than simply accumulating negative expressions.

[0040] Furthermore, the instantaneous emotion item includes any one or more of the following: emotion category, emotion polarity, emotion intensity, triggering context, original text evidence fragment, location information, and instantaneous emotion item confidence level;

[0041] The location information includes any one or more of the following: sentence number, question-and-answer round number, character start and end positions, and timestamp;

[0042] The stable features include any one or more of the following: trait type, persistence estimate, reproducibility, cross-context consistency, a subset of instantaneous sentiment items that support the corresponding stable features, trait score, and confidence of stable features.

[0043] Furthermore, the method for establishing a directional evidence path from transient emotion terms to stable feature terms includes:

[0044] Vector encoding is performed on each instantaneous emotion item and stable feature item to obtain instantaneous emotion vector and stable feature vector;

[0045] Based on the feature similarity between the instantaneous emotion vector and the stable feature vector, the contextual semantic consistency, temporal recurrence consistency, and evidence uncertainty between the instantaneous emotion item and the corresponding stable feature item are obtained;

[0046] Calculate the evidence relevance score and evidence relevance weight;

[0047] Based on the evidence association weights and preset path construction conditions, directional evidence paths are established from each instantaneous emotion item to a stable feature item; each directional evidence path includes at least: instantaneous emotion item, stable feature item, evidence association weights, and evidence relationship type; the evidence relationship type includes any one of support relationship, reverse relationship, and uncertain relationship.

[0048] To establish a computable association between transient sentiment items and stable feature items, this scheme performs vector encoding on both transient sentiment items and stable feature items, and calculates contextual semantic consistency, temporal recurrence consistency, and evidence uncertainty. Combining parameters such as the trait score of stable feature items, the confidence level of transient sentiment items, and the mapping similarity between the two levels of evidence vectors, an evidence association score is calculated and normalized to obtain the evidence association weight. Based on the evidence association weight and preset path construction conditions, a directional evidence path is established from each transient sentiment item to the stable feature item.

[0049] Furthermore, the method for enhancing the second feature and performing cross-modal feature interaction based on the hierarchical evidence chain includes:

[0050] The instantaneous emotion term set and the stable feature term set are serialized and vector-encoded respectively to obtain the instantaneous emotion representation and the stable feature representation;

[0051] The instantaneous emotion representation, stable feature representation, and second text features are fused together, and the fusion result is context-encoded to obtain text-emotion features;

[0052] Pooling is performed on the instantaneous emotion representation and stable feature representation, and the pooling result is input into the feedforward neural network to obtain emotional conditional cues;

[0053] Paired cross-attention is used to perform cross-modal interactions between text modality and audio modality, text modality and video modality, and audio modality and video modality to obtain cross-modal supplementary features; during the cross-modal interaction process, the modal features corresponding to the text modality adopt the text-emotion features, the modal features corresponding to the audio modality adopt the second audio features, and the modal features corresponding to the video modality adopt the second video features;

[0054] The features of each modality are fused with the corresponding cross-modal supplementary features to obtain the enhanced second feature.

[0055] After the hierarchical evidence chain is constructed, the transient emotion term set and the stable feature term set are used for fine-grained feature enhancement, rather than directly as the third feature. The transient emotion representation, the stable feature representation, and the second text feature are combined to form a text-emotion representation, and the transient emotion representation and the stable feature representation are used to generate emotional conditional cues, so that local state evidence and stable feature evidence can be incorporated into the subsequent cross-modal interaction process.

[0056] In this scheme, for any two different modalities among text, audio, and video modalities, the feature of the modality to be enhanced is used as the query, and the concatenation result of the feature of the other modality and the sentiment conditional prompt is used as the key feature and value feature. Cross-attention processing is then performed to obtain cross-modal supplementary features. When the modality to be enhanced is text, the query feature is a text-sentiment feature; when the modality to be enhanced is audio or video, the corresponding second audio feature or second video feature is used. Of course, if the modality mask corresponding to the modality providing the cross-modal supplementary feature is invalid, the cross-modal feature interaction path corresponding to that modality is skipped.

[0057] By employing pairwise interactions between text-audio, text-video, and audio-video pairs, and combining them with modal masks for fusion, enhanced second text features, second audio features, and second video features are obtained. In this fusion, the threshold is obtained by concatenating the second feature of the modality to be enhanced with its cross-modal supplementary features, followed by linear mapping and a sigmoid function, and then weighting the two feature sets according to the threshold.

[0058] Unlike the transient emotion and stable feature terms, which are used to enhance the second feature, the evidence reasoning term alone serves as a high-level evidence representation. The supporting evidence, counter-evidence, evidence uncertainty, emotion-cognition-behavioral correlation information, and directional evidence paths in the evidence reasoning term are serialized and vectorized to obtain the third feature, thus distinguishing fine-grained evidence enhancement from high-level evidence reasoning at the feature level.

[0059] Furthermore, the evidence-guided fusion module includes:

[0060] Feature aggregation unit: used to perform pooling or temporal aggregation on the text features, audio features and video features in the enhanced second feature to obtain text modality-level features, audio modality-level features and video modality-level features;

[0061] Consistency calculation unit: used to obtain global evidence consistency based on the mapping similarity between the text modal features, audio modal features, and video modal features and the third feature; and to pool the instantaneous sentiment representation and the stable feature representation respectively, calculate the mapping similarity between the text modal features, audio modal features, and video modal features and the pooled instantaneous sentiment representation and the pooled stable feature representation respectively, and take the average value to obtain local evidence consistency;

[0062] Weight calculation unit: used to calculate the evidence fusion score, evidence-guided fusion weight, and multimodal fusion features based on the global evidence consistency and local evidence consistency;

[0063] The fourth feature module is used to concatenate and fuse the multimodal fusion feature with the third feature, perform cross-attention fusion, or gated fusion to obtain the fourth feature.

[0064] In this scheme, the evidence-guided fusion module aggregates the enhanced text, audio, and video features to obtain modal-level features. Global evidence consistency measures the degree of matching between each modal-level feature and the overall evidence semantics represented by the third feature; local evidence consistency measures the degree of matching between each modal-level feature and the transient sentiment representation and stable feature representation. Further evidence fusion of the multimodal fused features and the third feature yields the fourth feature.

[0065] Furthermore, the classification explanation module generates an interpretability report using the following method:

[0066] Calculate the category association weights based on the stable feature vectors corresponding to each stable feature item and the classification results;

[0067] Based on category association weights, the evidence contribution of each directional evidence path to each predefined depressive state category is calculated;

[0068] The evidence paths in each direction are sorted according to their contribution to the evidence and the corresponding original text fragments are retrieved.

[0069] Establish a backtracking path of "stable feature item - instantaneous sentiment item - original text fragment", and generate an interpretability report based on the backtracking path.

[0070] After determining the target category, this scheme further calculates the category association weight between stable features and the target category, and obtains the evidence contribution of each directional evidence path to the target category.

[0071] Furthermore, the process of sorting the directional evidence paths according to their contribution and retrieving the corresponding original text fragments includes:

[0072] The directional evidence paths are classified into supporting evidence paths, opposing evidence paths, and uncertain evidence paths;

[0073] The supporting evidence paths, reverse evidence paths, and uncertain evidence paths are sorted separately.

[0074] Based on the sentence number, question-and-answer round, character position, and timestamp saved for the instantaneous sentiment item corresponding to each directional evidence path, the corresponding original text fragments are retrieved respectively.

[0075] In this scheme, the classification and interpretation module sorts the supporting evidence path, the reverse evidence path, and the uncertain evidence path according to the contribution of the evidence, and retrieves the corresponding original text fragments according to the location mapping of the instantaneous sentiment item, and generates an evidence-based interpretability report along the path of "stable feature item - instantaneous sentiment item - original text fragment".

[0076] Therefore, in this application, the large language model is used to construct structured hierarchical evidence and evidence relationships, and does not directly output the final depression state category; the final classification result is determined by inputting the fourth feature into the classification model.

[0077] Compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0078] 1. This invention can extract, associate, and hierarchically aggregate low-intensity emotional cues scattered in different text contexts, reducing the interference of emotional ambiguity on the identification of depressive states, thereby improving the model's recognition accuracy.

[0079] 2. This invention designs an evidence-guided fusion module, which adaptively adjusts the fusion weights of text, audio and video modalities by integrating the global and local consistency between different modal features and high-level semantic information, thereby enhancing the effective synergy between different modal information and improving the effectiveness of multimodal information fusion and the performance of depression state recognition.

[0080] 3. By establishing directional association paths and combining the mapping relationship between each feature item and the original text position, this invention enables the classification results to be traced back to the corresponding stable features, instantaneous emotions and original text fragments, thereby improving the traceability and interpretability of the depressive state identification results. Attached Figure Description

[0081] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0082] Figure 1 This is a schematic diagram of the system framework of a specific embodiment of the present invention;

[0083] Figure 2 This is a schematic diagram of the system operation flow according to a specific embodiment of the present invention;

[0084] Figure 3 This is a flowchart illustrating the process of constructing a hierarchical chain of evidence in a specific embodiment of the present invention. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0086] Example 1:

[0087] like Figure 1 and Figure 2 The multimodal depression recognition system based on evidence reasoning using a large language model, as shown, includes:

[0088] Multimodal data processing module: used to acquire multimodal interaction data of the target object, extract features and perform time correspondence on the multimodal interaction data to obtain a first feature; the multimodal interaction data includes text data, audio data and video data;

[0089] Feature encoding module: used to encode the first feature to obtain the second feature;

[0090] Evidence reasoning module: used to construct a hierarchical evidence chain based on the text data in the multimodal interaction data through a large language model, enhance the second feature and perform cross-modal feature interaction based on the hierarchical evidence chain to obtain the enhanced second feature; also used to encode the evidence reasoning items in the hierarchical evidence chain to obtain the third feature;

[0091] Evidence-guided fusion module: used to perform feature aggregation on the enhanced second feature, determine the evidence-guided fusion weight, and obtain the fourth feature based on the evidence-guided fusion weight and the third feature;

[0092] The classification and interpretation module is used to input the fourth feature into the depression state classification model, obtain the classification result, and generate an interpretable report.

[0093] The system in this embodiment is implemented by a computer device, which includes a processor and a memory. When the processor executes the program in the memory, it completes the data processing corresponding to the above five modules: the multimodal data processing module forms a first feature, and the feature encoding module forms a second feature; the evidence reasoning module uses transient sentiment terms and stable feature terms to form an enhanced second feature, and encodes the evidence reasoning terms to form a third feature; the evidence-guided fusion module combines the enhanced multimodal representation with the third feature to form a fourth feature; and the classification and interpretation module completes the classification based on the fourth feature and backtracks the original text along the directional evidence path.

[0094] Specifically:

[0095] (a) Multimodal data processing module, including:

[0096] Text processing unit: used to perform sentence segmentation, word segmentation, and linguistic information extraction on the text data to obtain first text features;

[0097] Audio processing unit: used to resample, segment, denoise, and extract acoustic features from the audio data to obtain the first audio feature;

[0098] Video processing unit: used to perform face detection, facial key point localization, and visual behavior feature extraction on the video data to obtain the first video features.

[0099] Time correspondence unit: used to correspond the first text feature, the first audio feature, and the first video feature to the same interview segment or time window based on the timestamps of the text data, audio data, and video data.

[0100] The multimodal data processing module acquires text, audio, and video data from the same interview or question-and-answer process, and performs preprocessing and feature extraction on the three modalities respectively.

[0101] For text data, the text processing unit uses NLTK to perform sentence segmentation, word segmentation, and linguistic information extraction to obtain words, parts of speech, keywords, and emotional and cognitive-related linguistic information, thereby forming the first text feature.

[0102] For audio data, the audio processing unit uses LibROSA to perform resampling, speech framing and noise reduction, and extracts Mel frequency cepstral coefficients, pitch, energy, speech rate, pauses and prosodic features, thereby forming the first audio features.

[0103] For video data, the video processing unit uses OpenFace to perform face detection and facial key point analysis, and extracts facial motion units, gaze direction, head posture and expression change features, thereby forming the first video features.

[0104] The time-correspondence unit maps the three primary features to the same interview segment or time window based on the timestamps of the text fragments, audio frames, and video frames, and generates a modality mask based on the valid status of the text, audio, and video data within each time window; the mask is set to 1 when the modality is valid and 0 when the modality is missing.

[0105] (ii) Feature encoding module, including:

[0106] Text encoder: used to encode the first text features;

[0107] Audio timing encoder: used to encode the first audio feature;

[0108] Video temporal encoder: used to encode the first video features;

[0109] Feature mapping unit: used to map the encoding results of text encoder, audio temporal encoder and video temporal encoder to a unified dimension, to obtain the second text feature, the second audio feature and the second video feature respectively, and combine them to obtain the second feature.

[0110] The feature encoding module performs contextual or temporal encoding on the first text feature, the first audio feature, and the first video feature, respectively.

[0111] For the text modality, the text encoder uses the BERT encoder to perform contextual encoding on the first text features to obtain a text representation that includes semantic dependencies.

[0112] For the audio and video modalities, the audio temporal encoder and the video temporal encoder respectively use the Transformer encoder to temporally encode the first audio feature and the first video feature to model the acoustic and visual behavioral changes during the interview process.

[0113] The feature mapping unit performs linear feature mapping on the above three encoding results, so that the text, audio and video representations have a unified feature dimension, and obtains the second text feature, the second audio feature and the second video feature.

[0114] The encoding and mapping process is represented as follows:

[0115] ;

[0116] In the formula: These represent the second text feature, second audio feature, and second video feature after unifying the dimensions, respectively. These represent linear feature mappings for text, audio, and video features, respectively. For BERT encoders, and For the corresponding Transformer encoder; , Let represent the first text feature, first audio feature, and first video feature of the i-th target object, respectively; where, Indicates the first The second modal feature corresponding to the modality of each target object; Indicates the length of the corresponding feature sequence; This represents the feature dimensions after a unified mapping of text, audio, and video.

[0117] (III) Evidence Reasoning Module:

[0118] The evidence reasoning module constructs a hierarchical chain of evidence based on textual data. First, the complete interview is divided into multiple local textual contexts according to sentence boundaries, question-and-answer rounds, and the location of topic changes. For each context, sentence numbers, question-and-answer rounds, character start and end positions, and timestamps are retained. Local textual contexts limit the triggering scope of instantaneous emotions, while simultaneously establishing the locational relationship between evidence items and the original text.

[0119] In this embodiment, the specific method for constructing a hierarchical chain of evidence is as follows:

[0120] The text data in the multimodal interaction data is divided into several local text contexts;

[0121] The local text context and the first prompt template are input into a large language model. Based on the large language model, multiple transient emotion items are obtained to represent local short-term emotional states, resulting in a set of transient emotion items. In this process, the large language model extracts short-term emotional states directly triggered by the current event, topic, or expression, forming the set of transient emotion items. , This represents the j-th instantaneous emotion item for the i-th target object. Each instantaneous emotion item records the emotion category, emotion polarity, emotion intensity, triggering context, original text evidence fragment, location information, and instantaneous emotion item confidence, ensuring that the local emotion state corresponds to its original text location.

[0122] For example, the first prompt template may refer to: "You are a text emotion evidence extraction model. Please identify the content that reflects the short-term emotional state of the target subject based only on the context of the local interview text, without making a diagnosis of depression or making inferences based on information outside the text."

[0123] Subsequently, the text data and the set of transient emotion items from the multimodal interaction data, along with the second cue template, are input into a large language model. Based on the large language model, the recurrence, persistence, and cross-contextual consistency of each transient emotion item in different local text contexts are analyzed to obtain multiple stable feature items used to characterize stable depression-related traits, thus obtaining a stable feature item set. In this process, the stable feature item set is formed through the large language model. , This represents the k-th stable feature of the i-th target object. Reproducibility reflects the recurrence of relevant momentary emotions in multiple contexts, persistence reflects the continuation of the relevant state within the interview time range, and cross-contextual consistency reflects whether the same or similar emotional, cognitive, or behavioral tendencies are presented under different events or topics.

[0124] Stable features are not directly determined by a single instantaneous emotion item, but rather formed by multiple mutually supporting local evidence. Local emotions triggered by a specific single event and lacking cross-contextual recurrence are not directly elevated to stable features. Each stable feature item records trait type, persistence, recurrence rate, cross-contextual consistency, a subset of instantaneous emotion items supporting the stable feature item, trait score, and confidence level of the stable feature item.

[0125] For example, the second prompt template can be: "You are a stable emotional feature induction model. Please combine the input interview text and the extracted set of transient emotional items to analyze the recurrence, persistence, and cross-contextual consistency of multiple transient emotional items in different local text contexts, and summarize the relatively stable features from them. Stable features should not be inferred directly from a single isolated emotion, and a diagnosis of depression should not be made."

[0126] Then, the text data, the set of transient emotion items, and the set of stable feature items from the multimodal interaction data are input into a large language model. Based on the large language model, the evidentiary association between each transient emotion item and each stable feature item is determined, and a directional evidentiary path from transient emotion items to stable feature items is established to obtain the evidentiary reasoning items. This step involves inputting the complete text and the set of transient emotion items S... i Stable feature set T i The third prompt template is input into the large language model to determine the evidentiary relationship between the two levels of evidence and form the evidentiary reasoning term E. i .

[0127] Among them, the evidentiary relationships include supporting relationships, opposing relationships, and uncertain relationships. A supporting relationship indicates that the local instantaneous state provides positive support for the corresponding stable feature; an opposing relationship indicates that the relevant expression reflects retained interests, adaptive functions, or information contrary to the target stable feature; an uncertain relationship indicates that the local state has a clear event trigger, other reasonable explanations, or insufficient evidence strength.

[0128] Evidence Reasoning Further record information on the correlation between emotion, cognition, and behavior, as well as information derived from transient emotion items. Pointing to stable feature terms The directional evidence path. The directional evidence path is used to indicate the direction of evidence organization and feature transmission, and does not indicate medical causality.

[0129] In summary, constructing a hierarchical chain of evidence The process can be referenced. Figure 3 Its mathematical representation is: .

[0130] Preferably, the instantaneous emotion item includes any one or more of the following: emotion category, emotion polarity, emotion intensity, triggering context, original text evidence fragment, location information, and instantaneous emotion item confidence level.

[0131] Preferably, the location information includes any one or more of the following: sentence number, question-and-answer round number, character start and end positions, and timestamp.

[0132] Preferably, the stable feature includes any one or more of the following: trait type, persistence estimate, reproducibility, cross-context consistency, a subset of instantaneous sentiment items that support the corresponding stable feature, trait score, and stable feature confidence.

[0133] In a more preferred embodiment, the method for establishing a directional evidence path from transient sentiment items to stable feature items includes:

[0134] For instantaneous emotion items and stable characteristic terms Vector encoding is performed separately to obtain instantaneous emotion vectors. and stable eigenvectors The study calculates the contextual semantic consistency, temporal recurrence consistency, and evidence uncertainty between the two levels of evidence. Contextual semantic consistency measures the semantic matching degree between a given momentary sentiment item and a candidate stable feature item; temporal recurrence consistency measures the repeated distribution and time span of momentary sentiment items supporting the same stable feature item in different contexts; and evidence uncertainty characterizes the credibility reduction caused by event triggers, alternative explanations, or conflicting evidence.

[0135] Subsequently, the evidence association score is calculated by combining the trait scores of stable feature items, the confidence level of instantaneous sentiment items, contextual semantic consistency, temporal recurrence consistency, and evidence uncertainty. This score is then normalized to obtain the evidence association weight, which is used to determine the directional evidence path. In this embodiment, the evidence association score and evidence association weight are calculated according to the following formula:

[0136]

[0137]

[0138] In the formula: Indicates the evidence relevance score; Both represent learnable mapping matrices; d represents the feature dimension after the evidence vector is mapped. Indicates the index of the target object. Indicates the first An index of instantaneous sentiment items for each target object. Indicates the index of stable feature terms; Indicates the first The first target object Trait scores for each stable feature item; This represents the confidence level of the j-th instantaneous sentiment term; This indicates semantic consistency within the context; Indicates consistency in time reproducibility; This indicates uncertainty regarding the evidence; These are learnable parameters; Indicates the weight of evidence association; Only indicates the first The total number of instantaneous emotion items for each object; The index for normalized summation is denoted as . The first object A momentary emotional item, ; superscript The symbol is for stable trait types and does not indicate transpose; the T outside the parentheses in the formula indicates the transpose of the eigenvector obtained by linear mapping.

[0139] In this embodiment, each directional evidence path includes at least: instantaneous sentiment item, stable feature item, evidence association weight, and evidence relationship type; the evidence relationship type includes any one of support relationship, reverse relationship, and uncertain relationship.

[0140] In this embodiment, after the hierarchical evidence chain is constructed, the transient sentiment term set and the stable feature term set are used for feature layer enhancement, rather than directly as the third feature. The two levels of evidence are serialized and concatenated with the second text feature, and then input together into the BERT encoder to form a text-sentiment representation.

[0141] Specifically, the second feature is enhanced and cross-modal feature interaction is performed using the following method based on a hierarchical evidence chain:

[0142] The instantaneous emotion term set and the stable feature term set are serialized and vector-encoded respectively to obtain instantaneous emotion representation and stable feature representation; the instantaneous emotion representation and stable feature representation are fused with the second text feature, and the fusion result is context-encoded to obtain text-emotion feature.

[0143] The instantaneous emotion representation and stable feature representation are averaged and pooled. The pooling result is then input into a two-layer feedforward neural network to form an emotion conditional cue corresponding to the current sample.

[0144] Paired cross-attention is used to perform cross-modal interactions between text-audio, text-video, and audio-video to obtain cross-modal supplementary features. For example, the second feature of the modality to be enhanced (the m-th modality) is used. As a query, the second feature of another available modality (the nth modality) With emotional condition cues The concatenation result is used as the key and value to obtain cross-modal supplementary features. Modality masks conceal missing modalities before interaction. For the m-th modality, cross-modal supplementary features from the remaining available modalities are linearly mapped and aggregated, and then the aggregated result is combined with... After concatenation, the input linear layer and Sigmoid function are used to obtain the gate value, which is then used to pair the gate values. The enhanced second text feature, second audio feature, and second video feature are obtained by weighting the aggregated cross-modal supplementary features. The formula for calculating cross-attention is expressed as follows:

[0145] ;

[0146] In the formula: CA() represents the cross-attention operation; Indicates the first Among the target objects, the first one is... Modal orientation Cross-modal complementary features provided by the modality; Indicates the first The emotional condition cues corresponding to each target object; [] indicates feature concatenation operation; and ,in These represent text, audio, and video modalities, respectively.

[0147] When the modality to be enhanced is a text modality, the query feature uses text-sentiment features; when the modality to be enhanced is an audio modality or a video modality, the query feature uses the corresponding second audio feature or second video feature.

[0148] The evidence reasoning module is also used to encode evidence reasoning items in the hierarchical evidence chain to obtain the third feature. The specific process is as follows:

[0149] The supporting evidence, counter-evidence, evidence uncertainty, emotion-cognition-behavioral correlation information, and directional evidence path in the evidence reasoning terms are serialized and input into the Transformer evidence encoder to obtain the third feature. The mathematical representation of this process is as follows:

[0150] ;

[0151] In the formula: Indicates the third characteristic, Serialize(E i ) represents the serialization result of the evidence reasoning terms, Encoder E This represents the Transformer evidence encoder. Thus, the transient sentiment term and stable feature term primarily function to enhance the semantics of the second feature and facilitate cross-modal interaction, while the third feature focuses on representing the high-level support, reverse, and uncertain evidence semantics formed by the evidence reasoning term. These two aspects respectively undertake fine-grained feature enhancement and high-level evidence guidance, and combine during the evidence guidance fusion stage.

[0152] (iv) Evidence-guided fusion module:

[0153] The evidence-guided fusion module performs average pooling on the enhanced text, audio, and video features to obtain text modality-level features, audio modality-level features, and video modality-level features. Specifically, it includes a feature aggregation unit, a consistency calculation unit, a weight calculation unit, and a fourth feature module.

[0154] Feature aggregation unit: used to perform pooling or temporal aggregation on the text features, audio features and video features in the enhanced second feature to obtain text modality-level features, audio modality-level features and video modality-level features.

[0155] Consistency calculation unit: used to obtain global evidence consistency based on the mapping similarity between the text modality-level features, audio modality-level features, and video modality-level features and the third feature; and to pool the instantaneous sentiment representation and stable feature representation respectively, calculate the mapping similarity between the text modality-level features, audio modality-level features, and video modality-level features and the pooled instantaneous sentiment representation and pooled stable feature representation respectively, and take the average value to obtain local evidence consistency.

[0156] Among them, global evidence consistency is achieved through modal-level features and third features. The mapping similarity between them is obtained to measure whether the overall representation of a certain modality is consistent with the high-level evidence semantics summarized by the evidence reasoning terms.

[0157] Among them, the consistency of local evidence is denoted as By performing average pooling on both the instantaneous emotion representation and the stable feature representation, the cosine similarity between the m-th modal-level feature and the two pooled evidence representations after linear mapping is calculated, and the arithmetic mean of the two similarities is taken as the average. It is used to measure the degree of consistency between the modality and fine-grained evidence such as transient sentiment items and stable feature items.

[0158] Weight calculation unit: used to calculate the evidence fusion score, evidence-guided fusion weight and multimodal fusion features based on the global evidence consistency and local evidence consistency.

[0159] This embodiment combines global evidence consistency and local evidence consistency. Together with modality masks, they are used to calculate the evidence fusion score and the evidence-guided fusion weights. The first term in the evidence fusion score is the mapping similarity between modality-level features and the third feature, and the second term is a local evidence consistency weighting term. Subsequently, the modality mask is used to mask invalid modalities and normalize the results to obtain the evidence-guided fusion weights. The specific calculation formulas for the fusion weights and multimodal fusion features are as follows:

[0160] ;

[0161] In the formula: Let represent the evidence fusion scores corresponding to the m-th modality and the n-th modality, respectively; This represents the evidence-guided fusion weight corresponding to the m-th modality; Indicates multimodal fusion features, ; d represents the learnable mapping matrix; d represents the feature dimension after mapping. This represents a learnable parameter used to adjust the contribution of local evidence consistency. This indicates the consistency of local evidence corresponding to the m-th modality; , These represent the modality mask corresponding to the m-th mode and the modality mask corresponding to the n-th mode, respectively. The mask is set to 1 when the mode is available and 0 when the mode is unavailable, and the modality mask of at least one mode is 1. Weighting for consistency of local evidence; Denotes the set of modes, where These represent text modality, audio modality, and video modality, respectively. Indicates the first The first target object Modal-level features corresponding to a mode.

[0162] The fourth feature module is used to concatenate and fuse the multimodal fusion feature with the third feature, perform cross-attention fusion, or gated fusion to obtain the fourth feature. The mathematical expression for obtaining the fourth feature is:

[0163] ;

[0164] In the formula: The fourth feature is represented by LayerNorm; LayerNorm represents the layer normalization operation; and CA represents the aforementioned cross-attention operation.

[0165] It can be seen that the fourth feature simultaneously includes text, audio and video information after evidence-guided weighted aggregation, as well as high-level semantic information corresponding to evidence reasoning terms. It is a unified representation of multimodal information and hierarchical evidence information before decision-making, and serves as the direct input to the classification prediction layer.

[0166] (v) Classification and Explanation Module:

[0167] The classification interpretation module inputs the fourth feature into the fully connected classification prediction layer and obtains the classification probability of each preset depression state category through Softmax. The classification result is determined based on the maximum classification probability.

[0168] The classification and explanation module also generates interpretability reports using the following methods:

[0169] After obtaining the target category, first, based on stable feature terms... The corresponding stable feature vector The target category is represented, and the category association weight is calculated to describe the degree of association between each stable feature and the current target category.

[0170] Subsequently, the evidence association weights from transient emotion items to stable feature items were combined. Stable feature item trait score Confidence level of instantaneous emotion item Category association weight and the evidence direction coefficient Calculate the evidence contribution of each directional evidence path to the target category.

[0171] The evidence direction coefficient is determined by the type of evidence relationship and its confidence level: the confidence level of the evidence relationship is denoted as... Support relationship Reverse relation Uncertain Relationship and with a preset direction threshold Low-confidence paths are classified as uncertain paths. The calculation formula is as follows:

[0172] ;

[0173] In the formula: This represents the category association weight between the k-th stable feature and the c-th preset depression state category; W represents the contribution of the j-th instantaneous emotion term to the c-th presupposed depressive state category through the k-th stable feature term; p It is a learnable mapping matrix; This represents the stable eigenvector corresponding to the k-th stable feature term; This represents the category representation corresponding to the c-th preset depression state category; C represents the number of preset depression state categories. The evidence direction coefficient represents the corresponding directional evidence path.

[0174] when Greater than or equal to the preset direction threshold When, the corresponding directional evidence path is the supporting evidence path; when Less than or equal to a negative preset direction threshold When, the corresponding directional evidence path is the reverse evidence path; when The absolute value is less than the preset direction threshold. At that time, the corresponding directional evidence path is an uncertain evidence path.

[0175] In this embodiment, the confidence level of the evidence relationship output by the large language model is denoted as... ,and Supports corresponding relationships Pick The opposite relationship Pick Uncertain relation corresponding to Take 0.

[0176] The primary supporting paths, reverse paths, and uncertain paths are ranked according to their contribution to evidence. The corresponding original text fragments are retrieved based on the sentence number, question-and-answer rounds, character positions, and timestamps stored in the transient sentiment item. Therefore, the interpretability report not only provides the main stable features but also traces back the basis for their formation along the path of "stable feature item - transient sentiment item - original text fragment." The system ultimately outputs the classification results and the aforementioned report.

[0177] This embodiment generates a hierarchical evidence chain offline before training the classification network and fixes the corresponding structured evidence representation, while saving the mapping between each instantaneous sentiment item and the original text position. The classification network is trained using labeled multimodal samples; during inference, the same multimodal feature extraction, hierarchical evidence construction, evidence enhancement cross-modal interaction, third feature generation, evidence-guided fusion, and fourth feature classification are performed on the object to be identified, and evidence backtracking results are generated after classification is completed.

[0178] Example 2:

[0179] To verify the effectiveness of the method of the present invention in the task of identifying depressive states, this embodiment uses the DAIC-WOZ dataset for experiments. This dataset contains 189 subjects, divided into a training set of 107 subjects, a validation set of 35 subjects, and a test set of 47 subjects. A PHQ-8 score of 10 is used as the binary classification threshold for depressive states. Training, validation, and testing are all conducted on an independent basis for each subject.

[0180] In this embodiment, for modalities for which pre-extracted features have been provided in the DAIC-WOZ dataset, the pre-extracted features are used as the first modal features of the corresponding modality and directly input into the feature encoding module for encoding.

[0181] The experiment was implemented using Ubuntu 20.04 operating system, Python 3.10.9, and PyTorch 2.3.1. The hardware platform included an Intel Core i9-10850K CPU and an NVIDIA GeForce RTX 4090 GPU. The classification network used the AdamW optimizer, with an initial learning rate of 1×10⁻⁶. −4 The batch size was set to 6, the number of training rounds was set to 200, the text-sentiment representation dimension was set to 512, the audio and video encoder output dimension was set to 256, the multimodal Transformer was set to 2 layers, and the number of cross-attention heads was set to 8.

[0182] Transient sentiment and stable feature terms were generated offline using GPT-4o, while evidence reasoning terms were generated offline using Qwen3-Max. The generated structured evidence was fixed during the classification network training phase, and cross-entropy loss was used in the classification prediction layer.

[0183] The evaluation metrics used are Precision, Recall, and F1.

[0184] To verify the recognition performance of the method of this invention relative to different technical approaches, representative deep learning methods and LLM-assisted methods from the prior art were selected for comparative experiments. The listed methods adopted the DAIC-WOZ classification results reported in publicly available literature; those without directly corresponding indicators were indicated by "-". The comparative experimental results are shown in Table 1.

[0185] Table 1 Comparison of the depression state identification results of the method of the present invention and existing methods on the DAIC-WOZ dataset.

[0186]

[0187] As shown in Table 1, the Precision, Recall, and F1 of the method of this invention are 0.915, 0.948, and 0.931, respectively, with Recall and F1 being the highest among the listed methods. SMFL has a Precision of 0.940, higher than the 0.915 of the method of this invention, but its Recall and F1 are 0.840 and 0.890, respectively. These results indicate that the method of this invention, while maintaining high accuracy, improves the detection ability of depression samples and achieves a more balanced overall classification performance.

[0188] Example 3:

[0189] To analyze the hierarchical evidence chain and the role of each component in evidence-guided multimodal fusion in this application, ablation experiments were conducted under the condition that data partitioning, input features, classification prediction layers, and training parameters remained consistent.

[0190] The transient emotion term (TES), stable feature term (SDT), and evidence reasoning term (EBR) were removed to examine the independent contributions of Level 3 evidence. Furthermore, evidence-enhanced cross-modal interaction (ECMI), evidence-guided fusion (EGF), and complete hierarchical evidence chain (HEC) were removed to examine the overall effect of hierarchical evidence in cross-modal representation learning and fusion decision-making. The experimental results are shown in Table 2.

[0191] Table 2 Ablation Experiment Results of Key Components of this Application

[0192]

[0193] As shown in Table 2, removing any key component leads to a decrease in the recognition performance of the system. After removing TES, SDT, and EBR, the F1 scores are 0.904, 0.886, and 0.900, respectively, with the removal of SDT showing a more significant decrease, indicating that stable features formed across contexts have a strong effect on the recognition of depressive states. Removing ECMI and EGF further reduces the F1 score to 0.878 and 0.863, respectively, indicating that hierarchical evidence is not only used for textual semantic representation but also needs to influence multimodal representation through cross-modal interaction and evidence-guided fusion. Completely removing the hierarchical evidence chain results in the F1 score dropping to 0.838, the largest decrease, indicating that hierarchical evidence construction, cross-modal evidence enhancement, and evidence-guided fusion together constitute the main sources of performance improvement in this invention.

[0194] Example 4:

[0195] To illustrate the construction and tracing process of the hierarchical chain of evidence, this embodiment translates, de-identifies, and summarizes typical expressions in the public interview samples, and numbers the original text fragments as R1 to R8, as shown in Table 3.

[0196] Table 3 Examples of Original Interview Text Excerpts

[0197]

[0198] The text fragments listed in Table 3 cover typical content such as limited emotional expression, bereavement-related grief, sleep disturbances, and retained interests and adaptive functions. The system combines the semantics and context of each fragment to extract transient emotional items S1 to S8, and further summarizes them into stable feature items T1 to T4. The specific transformation relationship is shown in Table 4.

[0199] Table 4 Examples of Hierarchical Evidence Chain Transformation

[0200]

[0201] As shown in Tables 3 and 4, when "suppressing emotions" appears alone in R1, it only forms a local transient emotion item S1; R2 forms S2, and R4 forms S4. S1, S2, and S4 come from different expressions and contexts, and after recurrence, persistence, and semantic consistency analysis, they collectively support the stable feature item T1 "tendency to internalize emotions." This process demonstrates that stable features are not directly determined by a single negative word or a single transient emotion item, but are formed by multiple mutually supporting local pieces of evidence.

[0202] R3 reflects grief related to a recent bereavement and forms the transient emotional item S3. Guilt, self-blame, and regret in R4 form the transient emotional item S4. S3, due to its clear event trigger, is insufficient on its own to form a stable feature; combined with S4 and its corresponding context, S3 and S4 together support the stable feature T2, "bereavement-related emotional vulnerability," while preserving the uncertain relationship corresponding to the event-based explanation. R5 and R6 form S5 and S6 respectively, and together support T3, "sleep-related negative responses"; R7 and R8 form S7 and S8, together forming T4, "preserved adaptive functioning," as evidence contrary to the direction of depression risk.

[0203] Based on the above evidence, the evidence reasoning items simultaneously organize supporting paths, reverse paths, and uncertain paths. For example, it can form a supporting path "R1 / R2 / R4→S1 / S2 / S4→T1", a supporting / uncertain path "R3 / R4→S3 / S4→T2", a supporting / uncertain path "R5 / R6→S5 / S6→T3", and a reverse path "R7 / R8→S7 / S8→T4". Therefore, the numbering of the original text fragment R, the instantaneous sentiment item S, and the stable feature item T in the case is related to... Figure 3 S shown i T i The hierarchical relationship remains consistent.

[0204] During the feature transition process, the instantaneous emotion term set S i and the set of stable characteristic terms T i This is fused with second text features to form text-emotion features, and emotional conditional cues are generated to guide cross-modal interactions between audio, text, and video; Evidence reasoning term E i Separate encoding forms the third feature The enhanced modal features are then fused using evidence-guided fusion to form a multimodal fusion feature. Then it merges with the third feature to form the fourth feature. .

[0205] After classification, the system calculates the contribution of each directional evidence path based on the target category and traces back the corresponding original text using the location indices saved in R1 to R8, thereby outputting the classification results and an evidence-based interpretability report. This output is used for automated auxiliary identification and analysis of depressive states and does not replace professional clinical diagnosis.

[0206] Finally, this embodiment details the specific text content of the interpretability report, including the following (1) to (6):

[0207] (1) Identification result: depression.

[0208] (2) Comprehensive Explanation: The system comprehensively analyzes the target object's text, audio, and video multimodal information, and the classification model ultimately identifies the object as depressed. The main supporting evidence related to this classification result focuses on emotional internalization, bereavement-related emotional vulnerability, and sleep-related negative reactions. Among them, emotional repression, limited emotional expression, guilt, self-blame, and regret appearing in multiple different interview contexts together form a relatively stable "tendency of emotional internalization," which constitutes an important supporting basis for this classification result. At the same time, the target object has recent bereavement-related grief and guilt and self-blame related to family events, and the above evidence together support "bereavement-related emotional vulnerability." However, since some grief expressions have clear recent bereavement events as triggers, the system retains this part of the evidence as uncertain evidence, and does not directly use bereavement grief as the basis for determining the state of depression. S3 formed by R3 must be combined with S4 to jointly support T2, and S3 itself is not enough to form a stable feature. In addition, sleep difficulties and irritability after sleep deprivation together form "sleep-related negative reactions," which support the depression category; however, since sleep difficulties are also accompanied by irregular working hours as an alternative explanation, this evidence path also retains a certain degree of uncertainty. On the other hand, the target subjects still exhibited travel interest, willingness to engage in social interaction, and ability to complete work tasks, thus forming "retained adaptive functions." This characteristic, which is opposite to the direction of depression risk, serves as reverse evidence in calibrating the classification results. After synthesizing all supporting evidence, reverse evidence, and uncertain evidence, the classification model finally outputs a depression category.

[0209] (3) The main evidence and retrospective results are shown in Table 5:

[0210] Table 5. Key Evidence and Retrospective Results

[0211]

[0212] (4) Overview of the evidence path:

[0213] R1 / R2 / R4 → S1 / S2 / S4 → T1 (Supported);

[0214] R3 / R4 → S3 / S4 → T2 (Supported / Uncertain);

[0215] R5 / R6 → S5 / S6 → T3 (Supported / Uncertain);

[0216] R7 / R8 → S7 / S8 → T4 (reverse direction).

[0217] (5) Key original evidence:

[0218] T1 Emotional Internalization Tendency: R1: "I often suppress a lot of emotions and am reluctant to express them." R2: "My family relationships are okay, but we are relatively closed off and rarely talk about our emotions directly." R4: "When my family went through difficult times, I felt guilty and regretful because I should have given them more care and support."

[0219] The three different contexts mentioned above form S1, S2 and S4 respectively, and together support T1, rather than forming a stable feature directly based on a single negative expression.

[0220] T2 Bereavement-related emotional vulnerability: R3: "My uncle recently passed away, and I've been feeling quite sad lately." R4: "...I feel some guilt and regret about it." Both pieces of evidence support T2, but R3 has a clear event trigger, therefore it is also marked as uncertain.

[0221] T3 Sleep-related negative responses: R5: "Because of my irregular work hours, I often have difficulty getting good sleep." R6: "When I don't sleep well, I become more irritable than usual." Both of these constitute sleep-related negative responses, while retaining the alternative explanation of "irregular work hours."

[0222] T4 retains adaptive functions: R7: "I enjoy traveling and meeting different people and experiencing different cultures." R8: "I recently completed a relatively important website project and received recognition from others." Both items indicate that interests, social engagement, and work abilities are still retained, thus serving as reverse evidence.

[0223] (6) Output conclusion:

[0224] System identification result: Depression. The main supporting evidence includes a tendency for emotional internalization across context, bereavement-related emotional vulnerability, and sleep-related negative reactions; some of this evidence is influenced by specific life events and work factors, thus retaining some uncertainty. Meanwhile, the retention of interests, social participation, and work ability constitutes reverse evidence. The system synthesizes the above evidence with multimodal features to arrive at the current classification result.

[0225] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0226] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

Claims

1. A multimodal depression recognition system based on evidence reasoning using a large language model, characterized in that, include: Multimodal data processing module: used to acquire multimodal interaction data of the target object, extract features and perform time correspondence on the multimodal interaction data to obtain the first feature; The multimodal interaction data includes text data, audio data, and video data; Feature encoding module: used to encode the first feature to obtain the second feature; Evidence reasoning module: used to construct a hierarchical evidence chain based on the text data in the multimodal interaction data through a large language model, enhance the second feature and perform cross-modal feature interaction based on the hierarchical evidence chain to obtain the enhanced second feature; also used to encode the evidence reasoning items in the hierarchical evidence chain to obtain the third feature; Evidence-guided fusion module: used to perform feature aggregation on the enhanced second feature, determine the evidence-guided fusion weight, and obtain the fourth feature based on the evidence-guided fusion weight and the third feature; The classification and interpretation module is used to input the fourth feature into the depression state classification model to obtain the classification result. It also generates an interpretability report.

2. The multimodal depression recognition system based on large language model evidence reasoning according to claim 1, characterized in that, The multimodal data processing module includes: Text processing unit: used to perform sentence segmentation, word segmentation, and linguistic information extraction on the text data to obtain first text features; Audio processing unit: used to resample, segment, denoise, and extract acoustic features from the audio data to obtain the first audio feature; Video processing unit: used to perform face detection, facial key point localization, and visual behavior feature extraction on the video data to obtain the first video features. Time correspondence unit: used to correspond the first text feature, the first audio feature, and the first video feature to the same interview segment or time window based on the timestamps of the text data, audio data, and video data.

3. The multimodal depression recognition system based on large language model evidence reasoning according to claim 2, characterized in that, The feature encoding module includes: Text encoder: used to encode the first text features; Audio timing encoder: used to encode the first audio feature; Video temporal encoder: used to encode the first video features; Feature mapping unit: used to map the encoding results of text encoder, audio temporal encoder and video temporal encoder to a unified dimension, to obtain the second text feature, the second audio feature and the second video feature respectively, and combine them to obtain the second feature.

4. The multimodal depression recognition system based on large language model evidence reasoning according to claim 3, characterized in that, The evidence reasoning module constructs a hierarchical chain of evidence using the following method: The text data in the multimodal interaction data is divided into several local text contexts; The local text context is input into the large language model, and multiple instantaneous emotion items are obtained based on the large language model to represent local short-term emotional states, resulting in a set of instantaneous emotion items; The text data in the multimodal interaction data and the set of transient emotion items are input into a large language model. Based on the large language model, the recurrence, persistence and cross-context consistency of each transient emotion item in different local text contexts are analyzed to obtain multiple stable feature items used to characterize stable depression-related traits, and thus obtain a set of stable feature items. The text data, instantaneous sentiment item set, and stable feature item set in the multimodal interaction data are input into the large language model. Based on the large language model, the evidence association relationship between each instantaneous sentiment item and each stable feature item is determined, and a directional evidence path from the instantaneous sentiment item to the stable feature item is established to obtain the evidence reasoning item. The set of instantaneous emotion items, the set of stable feature items, and the set of evidence reasoning items are used as the hierarchical evidence chain.

5. A multimodal depression recognition system based on large language model evidence reasoning according to claim 4, characterized in that, The instantaneous emotion item includes any one or more of the following: emotion category, emotion polarity, emotion intensity, triggering context, original text evidence fragment, location information, and instantaneous emotion item confidence level; The location information includes any one or more of the following: sentence number, question-and-answer round number, character start and end positions, and timestamp; The stable features include any one or more of the following: trait type, persistence estimate, reproducibility, cross-context consistency, a subset of instantaneous sentiment items that support the corresponding stable features, trait score, and confidence of stable features.

6. A multimodal depression recognition system based on large language model evidence reasoning according to claim 4, characterized in that, The method for establishing a directional evidence path from instantaneous sentiment terms to stable feature terms includes: Vector encoding is performed on each instantaneous emotion item and stable feature item to obtain instantaneous emotion vector and stable feature vector; Based on the feature similarity between the instantaneous emotion vector and the stable feature vector, the contextual semantic consistency, temporal recurrence consistency, and evidence uncertainty between the instantaneous emotion item and the corresponding stable feature item are obtained; Calculate the evidence relevance score and evidence relevance weight; Based on the evidence association weights and preset path construction conditions, directional evidence paths are established from each instantaneous emotion item to a stable feature item; each directional evidence path includes at least: instantaneous emotion item, stable feature item, evidence association weights, and evidence relationship type; the evidence relationship type includes any one of support relationship, reverse relationship, and uncertain relationship.

7. A multimodal depression recognition system based on large language model evidence reasoning according to claim 4, characterized in that, The method for enhancing the second feature and performing cross-modal feature interaction based on the hierarchical evidence chain includes: The instantaneous emotion term set and the stable feature term set are serialized and vector-encoded respectively to obtain the instantaneous emotion representation and the stable feature representation; The instantaneous emotion representation, stable feature representation, and second text feature are fused together, and the fusion result is context-encoded to obtain text-emotion features; Pooling is performed on the instantaneous emotion representation and stable feature representation, and the pooling result is input into the feedforward neural network to obtain emotional conditional cues; Paired cross-attention is used to perform cross-modal interactions between text modality and audio modality, text modality and video modality, and audio modality and video modality to obtain cross-modal supplementary features; during the cross-modal interaction process, the modal features corresponding to the text modality adopt the text-emotion features, the modal features corresponding to the audio modality adopt the second audio features, and the modal features corresponding to the video modality adopt the second video features; The features of each modality are fused with the corresponding cross-modal supplementary features to obtain the enhanced second feature.

8. A multimodal depression recognition system based on large language model evidence reasoning according to claim 7, characterized in that, The evidence-guided fusion module includes: Feature aggregation unit: used to perform pooling or temporal aggregation on the text features, audio features and video features in the enhanced second feature to obtain text modality-level features, audio modality-level features and video modality-level features; Consistency calculation unit: used to obtain global evidence consistency based on the mapping similarity between the text modal features, audio modal features, and video modal features and the third feature; and to pool the instantaneous sentiment representation and the stable feature representation respectively, calculate the mapping similarity between the text modal features, audio modal features, and video modal features and the pooled instantaneous sentiment representation and the pooled stable feature representation respectively, and take the average value to obtain local evidence consistency; Weight calculation unit: used to calculate the evidence fusion score, evidence-guided fusion weight, and multimodal fusion features based on the global evidence consistency and local evidence consistency; The fourth feature module is used to concatenate and fuse the multimodal fusion feature with the third feature, perform cross-attention fusion, or gated fusion to obtain the fourth feature.

9. A multimodal depression recognition system based on large language model evidence reasoning according to claim 4, characterized in that, The classification explanation module generates an interpretability report using the following method: Calculate the category association weights based on the stable feature vectors corresponding to each stable feature item and the classification results; Based on category association weights, the evidence contribution of each directional evidence path to each predefined depressive state category is calculated; The evidence paths in each direction are sorted according to their contribution to the evidence and the corresponding original text fragments are retrieved. Establish a backtracking path of "stable feature item - instantaneous sentiment item - original text fragment", and generate an interpretability report based on the backtracking path.

10. A multimodal depression recognition system based on large language model evidence reasoning according to claim 9, characterized in that, The process of sorting the directional evidence paths according to their contribution and retrieving the corresponding original text fragments includes: The directional evidence paths are classified into supporting evidence paths, opposing evidence paths, and uncertain evidence paths; The supporting evidence paths, reverse evidence paths, and uncertain evidence paths are sorted separately. Based on the sentence number, question-and-answer round, character position, and timestamp saved for the instantaneous sentiment item corresponding to each directional evidence path, the corresponding original text fragments are retrieved respectively.

Images