A depression assessment method based on heterogeneous multi-modal physiological behavior data

Abstract

The application relates to the technical field of artificial intelligence and mental health assessment, in particular to a depression assessment method and system based on heterogeneous multi-modal physiological behavior data. First, text, scales, audio, video, brain function and physiological data of a subject are synchronously acquired; after preprocessing, quality masks are generated, continuous features are extracted and feature post-processing is performed, and then the continuous features are discretized into structured pseudo-text token sequences; the token sequences and the quality masks are input into a unified embedding network, embedding coding, timing alignment, dynamic gating and cross-modal semantic fusion are sequentially performed, and a multi-modal data vector is output; the scale prompt tokens are jointly input into a large language model to perform conditional semantic modeling, and a semantic embedding matrix is obtained; finally, a multi-task learning model is used to simultaneously output depression assessment results and intervention effect prediction results. The application significantly improves the objectivity, robustness and explainability of the assessment, and is particularly suitable for early screening and individualized intervention guidance of adolescent depression.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and mental health assessment technology, and in particular to a depression assessment method based on heterogeneous multimodal physiological and behavioral data, as well as a computing device. Background Technology

[0002] Adolescent depression has become a prominent public health problem, characterized by early onset, wide-ranging impact, and close correlation with brain developmental plasticity. Early identification and intervention are crucial for improving prognosis. However, existing screening methods mainly rely on self-report scales and clinical interviews, which suffer from drawbacks such as strong subjectivity, susceptibility to subject reporting bias and assessor experience, and low efficiency in large-scale screening. In recent years, AI-based assessment methods have improved the objectivity of assessments to some extent, but significant limitations still exist.

[0003] Current technologies primarily rely on behavioral data, failing to incorporate objective indicators that directly reflect neural activity and physiological states, such as functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (FIR), electroencephalography (EEG), eye movement, and heart rate variability. This makes it difficult to comprehensively capture the deeper characteristics of adolescent depression, including abnormal brain connectivity, altered brain rhythms, and autonomic dysfunction. Furthermore, these methods do not consider data quality issues commonly found in real-world clinical or school screenings, such as head movement, electromyography (EMG), and motor artifacts, and lack mechanisms for handling low-quality or missing data, resulting in insufficient robustness of the models in real-world scenarios. In addition, existing methods are designed for general populations and have not been specifically optimized for the developmental plasticity of the adolescent brain, the age-related differences in physiological indicators, the characteristics of emotional expression, and the challenges of data collection.

[0004] Therefore, there is an urgent need for an intelligent assessment system that integrates multimodal brain function and physiological data, has data quality perception capabilities, is specifically designed for adolescents, and can simultaneously perform depression assessment and intervention effect prediction. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, this application provides a depression assessment method based on heterogeneous multimodal physiological and behavioral data, overcoming the following problems existing in the prior art:

[0007] First, objective physiological and brain function indicators are lacking. Existing methods mainly rely on scales and behavioral data such as audio, video, and text, failing to effectively integrate physiological indicators that can objectively reflect neural activity and autonomic nervous system status, such as functional magnetic resonance imaging, functional near-infrared spectroscopy, electroencephalography, eye movement, and heart rate variability. This makes it difficult to capture the deep-seated characteristics of adolescent depression, such as abnormal brain functional connectivity, changes in brain electrical rhythm, and autonomic nervous system dysfunction, thus limiting the objectivity and accuracy of the assessment.

[0008] Second, the quality differences and missing data in multimodal data have not been effectively addressed. During the data collection process, adolescents are prone to quality problems such as head movement, electromyography artifacts, motion artifacts, and blinking. Existing technologies have not designed corresponding processing mechanisms for the quality assessment indicators specific to each modality. Low-quality data is usually discarded directly or simply ignored, resulting in information waste and reducing the robustness of the model in real clinical or school screening scenarios.

[0009] Third, there is a lack of specific designs for adolescents. The adolescent brain is in a critical period of development, and its neurodevelopmental characteristics, emotional expression patterns, language abilities, and data collection challenges are significantly different from those of adults. Existing methods designed for general populations have not been optimized for these specificities, making it difficult to achieve optimal evaluation performance in adolescents.

[0010] Fourth, the task output is singular and cannot predict the intervention effect. Existing methods only output depression classification results and cannot predict an individual's possible response to interventions such as psychotherapy, medication, or behavioral activation. The adolescent brain has high plasticity, the early intervention window is short and the effects vary greatly, and simply providing a risk level is insufficient to meet the actual clinical needs for developing individualized intervention plans;

[0011] This application aims to overcome the shortcomings of the existing technologies mentioned above by integrating multimodal brain function and physiological data, designing a data quality perception mechanism, optimizing it specifically for the characteristics of adolescents, and using multi-task learning to simultaneously complete depression assessment and intervention effect prediction.

[0012] (II) Technical Solution

[0013] To achieve the above objectives, the main technical solutions adopted in this application include:

[0014] This application provides a method for depression assessment based on heterogeneous multimodal physiological and behavioral data, the specific steps of which include:

[0015] S1. Simultaneously acquire the subject's multimodal data and apply a unified timestamp and session identifier. The multimodal data includes at least text records, depression assessment scale data, audio data, video data, and brain function and physiological data. The brain function and physiological data includes at least functional magnetic resonance imaging data, functional near-infrared spectroscopy data, electroencephalogram data, eye movement data, and heart rate variability data.

[0016] S2. Preprocess the multimodal data respectively and generate a quality mask corresponding to each modality; the quality mask is used to identify the validity, missing segments and low-quality segments of each modality data segment; perform feature extraction on each preprocessed modality data to obtain continuous features of each modality data; perform feature post-processing operation on the continuous features; discretize the continuous features after feature post-processing using a level mapping method based on the quantile threshold of the training data, mapping the continuous values ​​to multiple level intervals, and generating a structured pseudo-text token sequence; wherein each token contains modality identifier, timestamp and feature level information;

[0017] S3. The structured pseudo-text token sequence and the quality mask are uniformly embedded into the network, and after performing embedding encoding, temporal alignment, dynamic gating and cross-modal semantic fusion in sequence, a multimodal data vector is output.

[0018] S4. Input the multimodal data vector and the scale cue token generated from the depression assessment scale data into the pre-trained large language model, use the scale cue token as a conditional cue to perform contextual semantic modeling, and output the semantic embedding matrix.

[0019] S5. Input the semantic embedding matrix into the multi-task learning model, and output the depression assessment results and intervention effect prediction results.

[0020] Optionally, in some embodiments of this application, the feature post-processing operation in S2 includes an age-adaptive feature normalization step:

[0021] The age information of the subjects is obtained and encoded to obtain an age feature vector;

[0022] Based on the age feature vector, scaling and translation parameters of the affine transformation are generated through a fully connected network.

[0023] Using the scaling and translation parameters, an affine transformation is performed on the continuous features to obtain age-normalized continuous features.

[0024] Optionally, in some embodiments of this application, the feature post-processing operation in S2 further includes a cross-modal missing completion step:

[0025] When any modality data is missing or its quality mask is marked as completely unavailable, a pre-trained cross-modal completion network is used to generate continuous features of the missing modality based on the continuous features of other available modalities, and the completion confidence is output.

[0026] The completed continuous features are used together with the original continuous features for subsequent discretization, and the completion confidence is used as an additional quality mask in the weight adjustment in subsequent cross-modal semantic fusion.

[0027] Optionally, in some embodiments of this application, the feature post-processing operation in S2 further includes a cross-modal consistency detection step:

[0028] The multimodal data is divided into several time-series segments based on the unified timestamp;

[0029] Within each time segment, the consistency of at least two modality pairs is calculated: the first pair is the emotional consistency between audio data features and video data features, and the second pair is the physiological arousal consistency between EEG data features and heart rate variability data features; the emotional consistency between audio data features and video data features is obtained by mapping them to a shared emotional embedding space and calculating the cosine similarity; the physiological arousal consistency between EEG data features and heart rate variability data features is obtained by calculating the Pearson correlation coefficient between the power of a specific frequency band of EEG data and the time-frequency domain index of heart rate variability data.

[0030] When the consistency of any pair of modalities is lower than a preset threshold, a collision mask for that modality is generated.

[0031] For each mode pair, the quality masks of the two modes in the group are averaged to obtain the local quality score of the group.

[0032] The conflict mask of the modality pair is fused with the local quality score to obtain the joint credibility of the modality pair in the time segment;

[0033] The joint confidence scores of all modality pairs within the same time segment are weighted and combined to obtain the overall confidence score of the time segment; when the overall confidence score is lower than a first preset threshold, it is determined to be a low-confidence segment.

[0034] The overall credibility and credibility determination results are associated with the pseudo-text tokens of the corresponding time segments to reduce the fusion weight of low-credibility segments in subsequent cross-modal semantic fusion.

[0035] Optionally, in some embodiments of this application, the embedding encoding in S3 includes:

[0036] A token embedding layer is used to map each pseudo-text token to a fixed-dimensional token vector;

[0037] A location embedding layer is used to add temporal location encoding to each token vector based on the unified timestamp and task anchor point;

[0038] A paragraph embedding layer is used to add a dimension encoding that identifies the modality of origin for each token vector;

[0039] The quality mask embedding layer is used to map the quality mask of each modality to a mask vector of the same dimension as the token vector through a linear transformation, and add or concatenate it with the token vector of the corresponding modality to obtain a token vector with quality information.

[0040] Optionally, in some embodiments of this application, the dynamic gating in S3 includes:

[0041] Calculate the overall quality score for each modality based on the quality mask of each modality data;

[0042] Completely block modal data whose overall quality score is lower than the second preset threshold;

[0043] Output the masked modal data and the reasons for masking, which are used to generate a data quality report.

[0044] Optionally, in some embodiments of this application, the cross-modal semantic fusion in S3 includes:

[0045] For modal data selected through dynamic gating, the fusion weight is determined in the attention mechanism based on the comprehensive quality score, wherein modal data with a higher comprehensive quality score receives a higher attention weight.

[0046] Optionally, in some embodiments of this application, a training step for the multi-task learning model is included before S1:

[0047] S0A: Obtain historical multimodal sample data labeled with depression status and intervention effect labels, and construct a training dataset;

[0048] S0B: Preprocess the training dataset to obtain a historical semantic embedding matrix; use the historical semantic embedding matrix to perform multi-task joint training on the initial multi-task learning model to obtain a multi-task learning model.

[0049] The initial multi-task learning model is based on a multi-task learning architecture, including a shared representation layer, a depression classification task head for outputting depression risk level or binary classification results, and an intervention prediction task head for outputting intervention effectiveness probability or response hierarchical labels.

[0050] Optionally, in some embodiments of this application, the multi-task joint training in the S0B includes:

[0051] The historical semantic embedding matrix is ​​input into the depression classification task head and the intervention prediction task head respectively, and the depression classification loss and the intervention prediction loss are calculated.

[0052] Based on the historical semantic embedding matrix, cross-modal consistency loss is calculated, which includes audio-video emotional consistency loss and EEG-heart rate variability physiological arousal consistency loss.

[0053] The depression classification loss, intervention prediction loss, and cross-modal consistency loss are weighted and summed, and the weights of each loss item are dynamically adjusted using an uncertainty weighting algorithm to obtain the total loss.

[0054] Based on the total loss, the parameters of the depression classification task head, the intervention prediction task head, and the shared representation layer are simultaneously updated using the backpropagation algorithm.

[0055] Optionally, in some embodiments of this application, the method further includes the following after S5:

[0056] At least one target modality token is selected from the semantic embedding matrix, and its discretization level is adjusted to a preset post-intervention target level to obtain the perturbed semantic embedding matrix.

[0057] The perturbated semantic embedding matrix is ​​re-inputted into the multi-task learning model to recalculate the depression assessment result and calculate the amount of change between the original depression assessment result and the original depression assessment result.

[0058] Based on the changes, the contribution of each modality feature to the depression assessment results is determined, and personalized intervention suggestion keywords are generated based on the contribution.

[0059] (III) Beneficial Effects

[0060] This application presents a depression assessment method based on heterogeneous multimodal physiological and behavioral data. By simultaneously collecting multimodal data such as text, audio, video, functional magnetic resonance imaging, functional near-infrared spectroscopy, electroencephalography, eye movement, and heart rate variability, it integrates behavioral characteristics with neurophysiological indicators, overcoming the shortcomings of traditional scale assessments, which are characterized by strong subjectivity and insufficient expression of single-modal information. By discretizing the features of each modality into a unified pseudo-text token sequence, a large language model can jointly model heterogeneous data in the same semantic space, effectively extracting deep cross-modal correlation features, thereby improving the objectivity and accuracy of depression risk assessment.

[0061] In the preprocessing stage, a quality mask is generated to identify the validity of data fragments. In the embedding and encoding stage, the quality mask is mapped to a mask vector with the same dimension as the token vector to participate in the attention fusion calculation. This enables the model to perceive the quality differences of data from different modalities and adaptively adjust the fusion weights, suppressing the impact of low-quality fragments such as head movement artifacts, electromyography interference, and motion artifacts on the evaluation results. This mechanism significantly improves the data robustness of the system in real clinical or campus screening scenarios.

[0062] To address the challenges of data collection and the unique characteristics of adolescent brain development plasticity and emotional expression, a multimodal data acquisition scheme incorporating brain function and physiological indicators was designed. A quality masking mechanism was employed to address data quality issues common in adolescents, such as head movements and artifacts. Compared to existing technologies designed for the general population, this approach more comprehensively captures neurodevelopmental abnormalities in adolescents, achieving superior assessment performance.

[0063] This application uses a large language model as a semantic fusion engine and combines scale cue tokens as conditional cues, enabling the model to output evidence citations and quality cues related to the assessment results during the reasoning process. This enhances the interpretability of the assessment results and makes it easier for clinicians to understand and trust the model's output. Attached Figure Description

[0064] Figure 1 This is a flowchart of a depression assessment method based on heterogeneous multimodal physiological and behavioral data, as described in this application.

[0065] Figure 2 This is a schematic diagram of the overall architecture of a multimodal large model-based adolescent depression risk assessment system in one embodiment of this application.

[0066] Figure 3 This is a schematic diagram of the confusion matrix of the model on the validation set in one embodiment of this application.

[0067] Figure 4 This is a schematic diagram of the receiver operating characteristic curve of the model on the validation set in one embodiment of this application. Detailed Implementation

[0068] To better explain and facilitate understanding of this application, a detailed description of its embodiments is provided below in conjunction with the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a clearer and more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0069] Example 1

[0070] Figure 1 According to this application, a method for assessing depression based on heterogeneous multimodal physiological and behavioral data is proposed, such as... Figure 1 As shown, this depression assessment method based on heterogeneous multimodal physiological and behavioral data includes:

[0071] S1. Simultaneously acquire the subject's multimodal data and apply a unified timestamp and session identifier. The multimodal data includes at least text records, depression assessment scale data, audio data, video data, and brain function and physiological data. The brain function and physiological data includes at least functional magnetic resonance imaging data, functional near-infrared spectroscopy data, electroencephalogram data, eye movement data, and heart rate variability data.

[0072] S2. Preprocess the multimodal data respectively and generate a quality mask corresponding to each modality; the quality mask is used to identify the validity, missing segments and low-quality segments of each modality data segment; perform feature extraction on each preprocessed modality data to obtain continuous features of each modality data; perform feature post-processing operation on the continuous features; discretize the continuous features after feature post-processing using a level mapping method based on the quantile threshold of the training data, mapping the continuous values ​​to multiple level intervals, and generating a structured pseudo-text token sequence; wherein each token contains modality identifier, timestamp and feature level information;

[0073] S3. The structured pseudo-text token sequence and the quality mask are uniformly embedded into the network, and after performing embedding encoding, temporal alignment, dynamic gating and cross-modal semantic fusion in sequence, a multimodal data vector is output.

[0074] S4. Input the multimodal data vector and the scale cue token generated from the depression assessment scale data into the pre-trained large language model, use the scale cue token as a conditional cue to perform contextual semantic modeling, and output the semantic embedding matrix.

[0075] S5. Input the semantic embedding matrix into the multi-task learning model, and output the depression assessment results and intervention effect prediction results.

[0076] This embodiment combines subjective behavioral information with objective neurophysiological indicators by simultaneously collecting and fusing text, scales, audio, video, and brain function and physiological data (fMRI, fNIRS, EEG, eye movement, HRV), overcoming the shortcomings of traditional scales that are highly subjective and lack sufficient single-modal information. By leveraging the deep semantic modeling capabilities of a large language model, it achieves unified representation and contextual association of cross-modal information, thereby improving the objectivity and accuracy of depression assessment.

[0077] By generating quality masks for each modality of data and performing dynamic gating and cross-modal semantic fusion based on the quality masks in a unified embedding network, the system can automatically identify and suppress the impact of low-quality, missing, or unreliable data fragments, thereby improving the system's adaptability to data quality fluctuations in real clinical or campus screening scenarios.

[0078] A multi-task learning model is adopted to simultaneously output depression assessment results and intervention effect prediction results, sharing a semantic representation layer, enabling the two tasks to promote each other and improving the model's generalization ability; at the same time, the intervention prediction results provide a reference for the development of individualized intervention programs, overcoming the shortcomings of existing technologies that only output classification results;

[0079] By using scale cue tokens as conditional cues to guide the large language model in contextual modeling, and combining quality masks and dynamic gating mechanisms, the model output can be accompanied by data quality reports and evidence clues, thereby improving the interpretability and clinical credibility of the assessment results.

[0080] Example 2

[0081] This embodiment provides a depression assessment method based on heterogeneous multimodal physiological and behavioral data, which can be applied to early screening and individualized intervention guidance for adolescent depression. Figure 2 This is a schematic diagram illustrating the overall architecture of a multimodal large model-based adolescent depression risk assessment system, as one embodiment of this application. The system mainly comprises four parts:

[0082] Multimodal data preprocessing module (corresponding) Figure 2 Top): Used for cleaning, correcting, and feature extraction of acquired fMRI, near-infrared, EEG, eye-tracking, physiological signals (heart rate), facial video, speech, and text data;

[0083] Unified Embedding and Encoding Module (corresponding) Figure 2 Upper middle section): Maps features from different modalities to a unified high-dimensional vector space and combines positional encoding to generate model input;

[0084] Core Reasoning and Evaluation Module (corresponding) Figure 2 Lower Middle Section: Utilizes pre-trained large language models (such as BERT, GPT, or Transformer architecture) to perform deep fusion and inference of multimodal features, and outputs depression risk prediction results;

[0085] Model validation and indicator analysis module (corresponding) Figure 2 Bottom): The classification performance of the model (such as recall, precision, and F1 score) is quantitatively evaluated using metrics such as confusion matrix, ROC curve, and AUC value.

[0086] The following details the specific steps.

[0087] 1. Synchronous acquisition and application of timestamps for multimodal data;

[0088] 1.1 Data Acquisition;

[0089] The following multimodal data from the subjects were acquired synchronously via a programmatic interface:

[0090] Textual records: The written content of the subjects' self-reports during the assessment process, as well as the text transcribed from their speech.

[0091] Depression assessment scale data: such as the PHQ-9 scale, recording the raw score for each item and the total score.

[0092] Audio data: Sampling frequency not less than 8kHz, saved as m4a or mp3 format, duration 1-5 minutes.

[0093] Video data: frame rate no less than 30fps, resolution no less than 480p, subject facing the camera.

[0094] Functional magnetic resonance imaging (fMRI) data: resting-state or task-state EPI sequences were used, with repetition time (TR) between 300 ms and 2000 ms, voxels of 3×3×3 mm³, and duration of 8 minutes.

[0095] Functional near-infrared spectroscopy (fNIRS) data: using resting-state and speech fluency task (VFT) sequences, 48 ​​channels, light source-detector spacing of 3 cm, sampling rate of 10 Hz or higher.

[0096] Electroencephalogram (EEG) data: 64 channels, sampling frequency not less than 256Hz, online reference FCz, recording electrooculogram (EOG).

[0097] Eye-tracking data: Sampling frequency not less than 60Hz, recording gaze point coordinates, pupil diameter and event markers.

[0098] Heart rate variability (HRV) data: RR interval sequences are obtained based on ECG or PPG.

[0099] 1.2 Unified timestamps and session identifiers;

[0100] All devices ensure start time alignment via clock synchronization protocols (such as NTP or hardware triggering). Upon data entry, each record is assigned a uniform timestamp (absolute or relative time based on the subject's start time in the assessment) and a session ID, along with the assessment task type (e.g., "resting state," "VFT task," "facial expression task") and segment number (e.g., which stimulus block in the task). This timestamp and session ID are used throughout all subsequent processing steps.

[0101] 2. Preprocessing and quality mask generation;

[0102] 2.1 Modal preprocessing;

[0103] Since the original formats, sampling rates, and noise characteristics of each modality differ significantly, they need to be preprocessed separately. The purpose of this preprocessing is to remove noise, correct artifacts, unify the basic format, and calculate quality-related indicators for each data unit.

[0104] Text and scales: Character encoding is unified to UTF-8, HTML tags are removed, word segmentation is performed, stop words are removed, and part-of-speech tagging is performed; scale items are scored in reverse, and dimensional scores and total scores are calculated.

[0105] Audio: Convert to 16kHz mono WAV; use Voice Activity Detection (VAD) to remove silence; pre-emphasis (0.97); frame segmentation (25ms window, 10ms shift), add Hamming window; spectral subtraction for noise reduction. Calculate signal-to-noise ratio, clipping rate, and speech percentage.

[0106] Video: Frame-by-frame decoding; face detection (RetinaFace), selection of the main target; face alignment (affine transformation); size normalization (224×224); pixel normalization (0-1). Calculate face detection confidence, sharpness (Laplacian variance), and illumination uniformity.

[0107] fMRI: DICOM was converted to NIfTI using dcm2niix; DPARSF (based on SPM12) was used for slice temporal correction (with intermediate slices as reference), head motion correction (outputting 6 parameters and frame shift FD), spatial registration to structural images, MNI spatial normalization (resampling 3×3×3 mm³); denoising regression (Friston-24 head motion parameters, cerebrospinal fluid, white matter signals); bandpass filtering (0.01–0.08 Hz); spatial smoothing (FWHM=6 mm). FD was calculated at each time point, and motion artifact frames were marked.

[0108] fNIRS: Homer3 is used to remove bad channels (low mean light intensity or high coefficient of variation); light intensity → light density; motion artifact detection (differential + sliding window standard deviation), using spline interpolation or wavelet denoising correction; bandpass filtering (0.01-0.2Hz); calculated using the modified Beer-Lambert law. [HbO]、 [HbR]. Marks the time points of motion artifacts.

[0109] EEG: Import using MNE; bandpass filtering (0.5-45Hz) + notch filtering (50Hz); bad conduction detection and interpolation; rereference (average reference); ICA removal of EOD, EMG, and ECG artifacts; segmented or sliding window; baseline correction. Record the proportion of bad conductions and the energy of artifact components.

[0110] Eye movement: Invalid samples are removed (valid=0 or coordinates outside the screen); short-term missing data (<75ms) linear interpolation; Savitzky-Golay smoothing; velocity thresholding is used to detect fixation / saccades / blinks. The valid sample rate is calculated.

[0111] HRV: ECG bandpass filtering (0.5-40Hz), Pan-Tompkins detection of R waves to obtain RR interval; PPG bandpass filtering (0.5-8Hz), detection of pulse peaks to obtain PP interval; abnormal intervals (<0.3s or >2.0s, or deviation from local median >30%) are removed, and spline interpolation is performed; resampling to 4Hz yields an equally spaced NN sequence. The proportion of abnormal intervals and effective heart rate are calculated.

[0112] 2.2 Unify timeline and segment division;

[0113] Although step 1 applied a uniform timestamp to the raw data, each modality may have undergone resampling, interpolation, and outlier removal during the basic preprocessing, resulting in the timeline no longer being perfectly aligned. Therefore, it is necessary to reconstruct the uniform timeline and segment the multimodal data.

[0114] (1) Unified timeline reconstruction;

[0115] A common timeline is established with the task start time as the origin (e.g., intervals of 0.1 seconds, from 10 seconds before the task starts to 30 seconds after the task ends). For each modality, its preprocessed data samples are mapped onto the common timeline based on their original timestamps: if samples exist near a certain time point, they are retained or interpolated; if there is no data for that modality at that time point (e.g., a long TR in fMRI), it is marked as missing. Through this step, all modalities obtain sequences of the same length with one-to-one time point correspondence.

[0116] (2) Multimodal segmentation;

[0117] Based on the task event anchors (event code, trigger time, segment number) recorded in step 1, the common timeline is divided into several temporal segments. Task anchor segmentation is preferred: for example, for verbal fluency tasks, the "baseline period (-10s to 0s)," "task period (0s to 60s)," and "recovery period (60s to 90s)" are each defined as a segment; for the Oddball task, a fixed window (e.g., -0.2s to 0.8s) is extracted centered on each stimulus presentation as a segment. For resting state or task-free data, a fixed time window (e.g., 2 seconds, 5 seconds, or 30 seconds, selected according to modality characteristics) is used for non-overlapping segmentation. Each segment retains a start timestamp, end timestamp, segment number, and task type label. After segmentation, the data subset within each segment for each modality can be used for subsequent feature extraction.

[0118] By unifying the timeline and segmenting, data from different modalities are strictly aligned in time, and the segment boundaries have clear physiological or behavioral significance, providing a unified analysis unit for cross-modal feature extraction and fusion.

[0119] 2.3 Generate a quality mask;

[0120] Based on the quality metrics obtained in step 2.1 of the preprocessing above (such as FD of fMRI, bad conduction ratio of EEG, signal-to-noise ratio of audio, effective sample rate of eye movement, etc.), a quality mask value ranging from [0,1] is generated for each data unit (frame, sample, time point, heartbeat) of each modality. Specific method:

[0121] For each modality, the quality metrics of all data units (frames, samples, heartbeats) within the segment are aggregated (e.g., the minimum or average value is taken) to obtain the segment-level quality score of the segment in that modality (the value ranges from 0 to 1, where 1 indicates complete reliability and 0 indicates complete unreliability).

[0122] If a modality is completely missing in a segment, its quality score is set to 0.

[0123] The quality scores of all modalities are combined into a vector, which serves as the quality mask for that segment.

[0124] Meanwhile, the quality mask can be output as a data quality report, providing clinicians with intuitive prompts on "which data are unreliable," enhancing the system's interpretability and clinical usability.

[0125] 2.4 Feature extraction;

[0126] Feature extraction is performed on each modal data within each time segment to obtain continuous numerical feature vectors. Specifically, this includes:

[0127] Text features: word frequency, TF-IDF, sentiment word density, negative word frequency, sentence length, etc.

[0128] Scale characteristics: scores for each item, scores for each dimension, total score, and scores for key items.

[0129] Audio features: fundamental frequency mean / standard deviation, energy, MFCC 1-13, speech rate, pause duration, etc. (each feature is statistically analyzed within the segment).

[0130] Video features: mean / peak AU intensity (e.g., AU01, AU04, AU12), mean / standard deviation of head pose Euler angles, and gaze direction dispersion.

[0131] fMRI features: ALFF and ReHo of ROI, and functional connectivity between ROI pairs (Pearson correlation coefficients after Fisher-z transform).

[0132] fNIRS characteristics: [HbO] and [HbR] represents the mean, peak value, peak area, rise slope, and latency within the task window.

[0133] EEG characteristics: absolute / relative power, center frequency, θ / β ratio, and frontal α asymmetry index in the δ, θ, α, β, and γ bands.

[0134] Eye movement characteristics: fixation duration, number of fixations, saccade amplitude, peak saccade velocity, mean / variation of pupil diameter, and scan path entropy.

[0135] HRV features: SDNN, RMSSD, pNN50, LF, HF, LF / HF, SD1, SD2, sample entropy.

[0136] All features are continuous numerical values, and the timestamp and modality identifier of the segment to which they belong are retained.

[0137] 2.5 Feature post-processing;

[0138] Feature post-processing includes three optional sub-steps (which can be selected to be performed depending on data quality):

[0139] 2.5.1 Age-adaptive normalization;

[0140] Adolescents are in a period of rapid development, and there are significant differences in the normal ranges of physiological indicators such as heart rate variability (HRV), brain electrical rhythm, and brain functional connectivity between 12-year-old and 18-year-old subjects. If the original characteristics are directly compared, the model can easily misjudge "lower heart rate variability due to younger age" as "autonomic dysfunction caused by depression," resulting in false positives.

[0141] This embodiment introduces age-adaptive normalization, using learnable affine transformations to calibrate the features of each subject to a uniform reference age scale. The specific steps are as follows:

[0142] (1) Obtain the subject’s age information (accurate to year or month) and encode it as an age feature vector (e.g., through sine / cosine position encoding or embedding layer).

[0143] (2) Construct an affine parameter generator consisting of a fully connected network, input the age feature vector, and output the scaling parameter γ and translation parameter β (each feature dimension corresponds to a γ and a β).

[0144] (3) Perform an affine transformation on each modality's continuous feature x, where x' is the age-normalized feature: We obtained the age-normalized features.

[0145] After normalization, healthy individuals of different ages exhibit similar characteristic distributions, while pathological deviations (such as abnormally low HRV due to depression) are still preserved. The model no longer needs to memorize "normal values" for different age groups, thus focusing on true pathological characteristics and significantly improving its generalization ability across age groups.

[0146] 2.5.2 Cross-modal missing data completion;

[0147] For children and adolescents, the total time spent remaining still to obtain usable fMRI data typically ranges from 30 minutes to 1 hour. Some children move their heads excessively, causing all data to be flagged as unusable. Discarding fMRI modalities directly results in the loss of significant information about brain functional connectivity related to depression, leading to decreased assessment accuracy or even making assessment impossible.

[0148] This embodiment designs a cross-modal missing data completion network, which uses other available modalities (such as EEG, HRV, eye-tracking, and behavioral data) to generate continuous features of the missing modality.

[0149] When a modal data is completely missing or its quality mask is marked as completely unavailable, the completion process is triggered, as follows:

[0150] (1) Concatenate the continuous features of other available modalities and input them into a pre-trained cross-modal completion network (e.g., based on conditional generative adversarial networks or multimodal variational autoencoders).

[0151] (2) The network outputs continuous features of the missing modalities and outputs a completion confidence score (0~1) to indicate the credibility of the generated features.

[0152] (3) The features generated by the completion are fed into the subsequent discretization step together with the original features; the completion confidence is used as an additional quality mask to participate in the subsequent attention weight adjustment (the weight of the completion feature with low confidence is reduced).

[0153] The cross-modal completion network is pre-trained independently based on historical samples with complete multimodal data, and optimized by minimizing the reconstruction error between generated features and real features. After pre-training, the parameters of the completion network are frozen and remain unchanged during the training of downstream depression classification tasks and multi-task learning models, without participating in gradient updates.

[0154] Even if fMRI is completely missing, the system can still infer approximate brain functional characteristics using signals such as EEG and HRV, ensuring that the assessment process is not interrupted. The completion confidence score is used to identify the reliability of the generated features. Based on this, the model reduces the decision weight of low-confidence completion features to avoid over-reliance on unreliable information.

[0155] 2.5.3 Cross-modal consistency detection;

[0156] Teenagers may deliberately conceal their depressive mood, saying "I'm fine" while displaying a sad facial expression and a low voice. Alternatively, anxiety may cause an increased heart rate and sweat gland secretion, leading to a mismatch between HRV and EEG arousal. Traditional unimodal or simple fusion methods cannot identify this discrepancy and may be misled by spoofed information, resulting in incorrect assessments.

[0157] This embodiment introduces cross-modal consistency detection, quantifies the degree of consistency between different modalities at the fragment level, and generates a conflict mask to reduce the impact of untrusted fragments. Specifically:

[0158] (1) Based on a unified timestamp, the multimodal data is divided into several time segments (each segment is usually 2-30 seconds).

[0159] (2) Within each segment, calculate the consistency of at least two mode pairs:

[0160] Audio-video sentiment consistency: Audio features (such as fundamental frequency, energy, MFCC) and video features (such as AU12 intensity, facial pleasure level) are mapped to a shared sentiment embedding space, and cosine similarity is calculated. The higher the similarity, the more consistent the sentiment expression.

[0161] EEG-HRV physiological arousal consistency: The Pearson correlation coefficient was calculated between EEG alpha power (relaxation / low arousal) and HRV LF / HF ratio (sympathetic / parasympathetic balance). Since increased relaxation (increased alpha power) is usually accompanied by decreased sympathetic activity (decreased LF / HF ratio) under normal physiological conditions, the two are negatively correlated. A positive or zero correlation indicates a conflict in physiological information.

[0162] (3) When the consistency of any pair of modal pairs is lower than the preset threshold (e.g., similarity < 0.5 or correlation coefficient < 0.2), generate a conflict mask for the modal pair (value range 0 to 1, 0 indicates severe conflict).

[0163] (4) For each mode pair, the quality masks of the two modes in the group are averaged to obtain the local quality score.

[0164] (5) The conflict mask is fused with the local quality score (e.g., multiplied) to obtain the joint confidence of the modality pair in this time sequence segment.

[0165] (6) The overall credibility of the segment is obtained by combining the joint credibility of all modal pairs within the same segment (e.g., taking the minimum value or weighted average). If the overall credibility is lower than the first preset threshold (e.g., 0.4), the segment is marked as a low credibility segment.

[0166] (7) The overall credibility is associated with the pseudo-text token of the corresponding time segment. In subsequent cross-modal semantic fusion, the fusion weight of low credibility segments is significantly reduced.

[0167] When subjects are "saying one thing but meaning another" or when nervousness leads to contradictory physiological signals, consistency testing can automatically identify unreliable segments and reduce their impact, effectively resisting spoofing and unintentional false positives, and improving the robustness and authenticity of the assessment. Simultaneously, this mechanism can output a "a segment has been downgraded due to emotional inconsistency" prompt, enhancing interpretability.

[0168] It should be noted that the aforementioned post-processing operations, such as age-adaptive normalization, cross-modal missing data completion, and cross-modal consistency detection, are not only applicable to adolescents but can also be extended to adults and even the elderly. For adults, age-adaptive normalization can eliminate normal physiological differences between 20 and 60 years old (such as a natural decrease in heart rate variability and weakened brain functional connectivity), avoiding misjudging normal aging as pathological changes; missing data completion and consistency detection have the same technical effect when processing adult data. However, this embodiment is specifically optimized to address three major practical challenges in adolescent depression assessment—rapid changes in physiological indicators during development, difficulties in acquiring brain imaging data, and the susceptibility of emotional expression to social desirability, leading to faking or avoidance. While these challenges also exist in adults, they are more prominent and clinically urgent in adolescence. Therefore, based on the general design, this method has been particularly enhanced to improve its adaptability to adolescents, making it a depression assessment solution suitable for all age groups, especially for early screening of adolescents.

[0169] 2.6 Discretize into a structured pseudo-text token sequence;

[0170] Continuous features after feature post-processing are converted into structured pseudo-text tokens that can be understood by a large language model. Each feature is discretized independently: a quantile-level mapping is used, based on the 20%, 40%, 60%, and 80% quantiles of the feature in the training dataset, mapping continuous values ​​to five levels from L1 to L5 (L1 being the lowest and L5 the highest). The quantile thresholds are pre-calculated and fixed.

[0171] Each token has a fixed format:

[0172] [MOD: Modal Identifier][SESSION: Session Identifier][TS: Timestamp] Feature Name: Level.

[0173] All tokens within the same time segment are grouped by modality, and within each group, they are arranged in lexicographical order by feature name; different segments are separated by segment boundary markers. The final generated structured pseudotext token sequence, along with the quality masks for each modality (original quality mask, completion confidence, collision mask, etc.), serves as the input for the next step.

[0174] The training dataset upon which the aforementioned quantile thresholds rely is historical multimodal sample data with the same distribution as the samples to be evaluated, pre-collected during the model training phase. Specifically, a sufficient number of samples are collected from the target population (e.g., adolescent students), each containing the same multimodal data (text, scales, audio, video, fMRI, fNIRS, EEG, eye-tracking, HRV) as S1, and labeled with depressive state and intervention effect labels. Preprocessing, feature extraction, and post-processing are performed on these training samples to obtain the distribution of each continuous feature, and then the 20th, 40th, 60th, and 80th quantiles of each feature are calculated as globally fixed quantile thresholds. In subsequent actual evaluations, these pre-calculated thresholds are directly used for discretization without recalculation. Once fixed, the quantile thresholds remain unchanged throughout the entire system deployment unless recalibrated using new training data.

[0175] 3. Unified embedded network processing;

[0176] 3.1 Embedded encoding;

[0177] like Figure 2 As shown, this embodiment sums and superimposes the vectors after token embedding, position embedding, paragraph embedding, and quality mask embedding to finally generate the input embedding shown in the figure. The input embedding in the figure is the final input vector after fusing modality type (paragraph embedding) and data quality information (quality mask embedding).

[0178] The structured pseudo-text token sequence and quality mask are uniformly embedded into the network. The embedding encoding consists of four sub-layers:

[0179] Token embedding layer: Maps each pseudo-text token to a fixed-dimensional vector (e.g., 768-dimensional).

[0180] Location embedding layer: Based on a unified timestamp and task anchor, a learnable temporal location code is added to each token vector.

[0181] Paragraph embedding layer: Add a learnable modality identifier vector (distinguishing between text, audio, video, fMRI, fNIRS, EEG, eye tracking, and HRV) to each token vector.

[0182] Quality Mask Embedding Layer: The quality masks of each modality (including the original quality mask, completion confidence, collision mask, etc.) are mapped to a mask vector of the same dimension as the token vector through a linear transformation, and then added element by element to the corresponding token vector to obtain a token vector with quality information.

[0183] 3.2 Timing Alignment;

[0184] Since different modalities have different sampling rates, the embedded vector sequences need to be aligned on the time axis. Specifically, based on the task event anchor points and unified timestamps from step 1, time resampling (downsampling high-sampling-rate modalities to the target frequency), sliding window aggregation (averaging multiple vectors within a window), or spline interpolation (interpolating missing time points) are used to generate a common time synchronization vector sequence for all modalities.

[0185] 3.3 Dynamic gating;

[0186] Based on the quality masks of each modality's data, calculate the overall quality score for each modality (e.g., the average or minimum value of the quality masks for all segments of that modality). Modality data with an overall quality score below a second preset threshold (e.g., 0.3) are completely masked and excluded from subsequent fusion. Simultaneously, the masked modality identifiers and the reasons for masking are output for generating a data quality report.

[0187] The second preset threshold is used for modal level gating determination.

[0188] 3.4 Cross-modal semantic fusion;

[0189] For modal data selected through dynamic gating, a multi-head attention mechanism is employed for fusion. Specifically, token vectors from different modalities are used as input, and fusion weights are assigned based on the overall quality score of each modality during attention calculation: modalities with higher quality scores receive higher attention weights (e.g., the quality score is used as a bias or scaling factor for the attention score). The output is a fused multimodal data vector with the same dimensions as the token vectors.

[0190] 4. Conditional reasoning based on large language models;

[0191] although Figure 2 The middle section schematically illustrates the feature extraction and fusion network, but in a preferred embodiment of this application, the core inference network employs a large language model based on the Transformer architecture.

[0192] The fused multimodal data vectors and scale cue tokens generated from depression assessment scale data are input into a pre-trained large language model (e.g., LLaMA, ChatGLM). The scale cue tokens are generated similarly to pseudo-text tokens: the total scale score and key item scores are discretized into level tokens and placed at the beginning of the sequence as conditional cues. The large language model uses a Transformer architecture encoder, decoder, or encoder-decoder structure for contextual semantic modeling. It utilizes an attention mechanism to fuse token vectors with quality mask information, outputting a semantic embedding matrix that integrates behavioral, brain function, physiological, and scale information.

[0193] Specifically, Figure 2 The "input embedding" layer shown actually transforms the "structured pseudo-text token sequence" (containing modality type, feature values, timestamps, and quality mask information) generated from multimodal data into a vector representation. Subsequently, these vectors undergo self-attention computation through a multi-layer Transformer encoder to capture long-distance dependencies between different modalities (such as EEG waveforms and facial micro-expressions), and finally output the depression risk assessment result through the Classification Head.

[0194] 5. Output and performance evaluation metrics of multi-task learning models;

[0195] The semantic embedding matrix is ​​input into a multi-task learning model. This model comprises a shared representation layer (e.g., a two-layer fully connected network), a depression classification task head, and an intervention prediction task head. The depression classification task head outputs a depression risk level (low / moderate / high) or a binary classification (depressed / non-depressed); the intervention prediction task head outputs the probability of intervention effectiveness (0-1) or a response hierarchical label (e.g., "likely effective," "likely ineffective"). The two task heads share the parameters of the shared representation layer and are jointly optimized during training.

[0196] To quantitatively evaluate the classification performance of the multi-task learning model, this embodiment introduces the following... Figure 2 The evaluation metrics are shown at the bottom. Specifically, the model determines the predicted category based on the output probability value and a preset threshold, and constructs a confusion matrix based on the predicted category and the true label. Recall, precision, accuracy, and F1 score are calculated based on the true positive (TP), true negative (TN), false positive (FP), and false negative (FN) statistics in the confusion matrix. Furthermore, the model's overall discriminative ability at different thresholds is further evaluated by plotting receiver operating characteristic (ROCCurve) curves and calculating the area under the curve (AUC).

[0197] 6. After the model is applied, counterfactual reasoning can be further conducted;

[0198] At least one target modality token (e.g., the "SDNN" token for HRV) is selected from the semantic embedding matrix, and its discretization level is adjusted to a predefined post-intervention target level (e.g., from L2 to L4, simulating improvement in heart rate variability). The perturbed semantic embedding matrix is ​​then re-input into the multi-task learning model to recalculate the depression assessment results and calculate the change from the original results. The larger the change, the greater the influence of the modality feature on the depression assessment. Based on the change of each modality, the contribution is determined, and personalized intervention suggestion keywords are generated (e.g., "It is recommended to increase aerobic exercise to improve heart rate variability").

[0199] This embodiment generates quality masks corresponding to each modality of data and applies them throughout the entire process of embedding encoding, dynamic gating, cross-modal semantic fusion, and even large language model inference, achieving unified perception and adaptive utilization of multimodal data quality. Specifically: ① In dynamic gating, low-quality modalities are directly masked to avoid noise contaminating the fusion results; ② In attention fusion, quality scores are used as the basis for weight allocation, achieving soft suppression of high-quality modalities dominating and low-quality modalities weakening; ③ The quality mask is passed to the large language model and multi-task learning model through the embedding layer, enabling subsequent semantic modeling and decision output to perceive and correct biases caused by unreliable data. Simultaneously, the quality mask can be output as a data quality report, providing intuitive credibility indicators.

[0200] This embodiment integrates objective physiological indicators and behavioral data such as fMRI, fNIRS, EEG, eye movement, and HRV, combined with deep semantic modeling of a large language model, overcoming the shortcomings of traditional scales, such as strong subjectivity and insufficient single-modal information. Through quality masking, dynamic gating, and quality weighting in cross-modal semantic fusion, it effectively suppresses the influence of low-quality segments such as head movements, artifacts, and missing data, making it particularly suitable for scenarios where adolescents have difficulty cooperating with high-quality data collection. The age normalization step eliminates normal physiological differences across age groups, making the model applicable to all age groups, especially important for adolescents in their developmental stage. By detecting audio-video emotional consistency and EEG... - HRV physiological arousal consistency generates conflict masks and reduces the weight of inconsistent segments, enabling the identification of subject faking or unintentional misreports and improving the reliability of the assessment. When a certain modality is completely missing, it uses other modalities to generate completion features, avoiding assessment interruption due to the absence of a single modality and improving the system's practicality. It simultaneously outputs depression assessment results and intervention effect prediction results, sharing semantic representations, with the two tasks mutually reinforcing each other. Intervention prediction results can assist clinicians in developing individualized treatment plans. Dynamically gating outputs data quality reports, counterfactual reasoning generates intervention suggestion keywords, and conditional prompts from the large language model can output evidence citations, making the assessment results transparent and reliable.

[0201] Example 3

[0202] This embodiment describes the training process of the multi-task learning model. It should be noted that the preprocessing of the training data follows the process of steps 2 to 4 in embodiment 2, that is, the historical multimodal sample data is sequentially subjected to modality preprocessing, unified time axis and segmentation, quality mask generation, feature extraction, feature postprocessing, and discretization into structured pseudo-text token sequences, and finally input into the large language model to obtain the semantic embedding matrix.

[0203] 1. Training data preparation;

[0204] A large amount of historical multimodal sample data was acquired, labeled with a depression status tag (e.g., a PHQ-9 total score ≥5 indicates depressive tendency, <5 indicates no depression) and an intervention effect tag (e.g., after 8 weeks of standardized psychotherapy, a PHQ-9 score reduction rate ≥50% is defined as effective intervention, otherwise it is ineffective). For each sample, the processing was strictly carried out according to steps S2 to S4 described in Example 2: first, modality preprocessing was performed, a unified time axis and segment division were implemented, and a quality mask was generated; then, continuous features of each modality were extracted, and post-processing operations such as age-adaptive normalization, cross-modal missing completion, and cross-modal consistency detection were performed; then, the continuous features were discretized into a structured pseudo-text token sequence, which was input into a large language model (using scale cue tokens as conditional cues) for contextual semantic modeling, and finally, the semantic embedding matrix of the sample was output. The semantic embedding matrices of all samples constitute the training dataset.

[0205] 2. Multi-task joint training;

[0206] Initialize the multi-task learning model, which includes the following components:

[0207] (1) Age-conditional shared representation layer: Age feature-wise linear modulation (Age-FiLM) is introduced in each layer. Specifically, the subject's age (accurate to the month) is encoded as a sinusoidal positional encoding vector a, and a scaling parameter γ is generated through a small fully connected network. l Translation parameter β l (One set for each feature dimension in each layer). The forward propagation of the shared representation is as follows: ;

[0208] Where l represents the layer index of the shared representation layer (l=1,2,…,L), and L is the number of layers in the fully connected network (L=2 in this embodiment). h0 represents the feature vector output of the (l-1)th layer (when l=1, h0 is the shared feature vector z of the input, which is a vector obtained by concatenating the semantic embedding matrix with the age code after global average pooling). This represents a fully connected linear transformation of the l-th layer (including the weight matrix and bias terms), which maps the input vector to the hidden dimension of that layer.

[0209] γ l (a) represents the scaling parameter vector of the l-th layer, generated by the age code a through a small fully connected network, with dimensions similar to Linear. l (h l−1 The same. This parameter controls the modulation of feature amplitude by age: features of subjects of different ages will be amplified or reduced to different degrees;

[0210] β l (a) represents the translation parameter vector of the l-th layer, which is also generated by the age code a through a small fully connected network, with dimensions similar to Linear. l (h l−1 The same. This parameter controls the offset of age from the feature baseline value; 'a' represents the age encoding vector, which is obtained by sinusoidal position encoding of the subject's age.

[0211] The core advantage of the Age-FiLM module lies in its ability to dynamically adjust feature representations to adapt to the differences in physiological baselines at different developmental stages by adaptively modulating the feature transformations of each layer based on age. This prevents the model from misinterpreting normal age-related changes as pathological features.

[0212] (2) Depression classification task head (binary classification, output depression probability);

[0213] (3) Intervention prediction task head (binary classification, outputting the probability of effective intervention);

[0214] (4) Optional auxiliary task head: Age regression task head (used for regularization, only used during the training phase, removed during inference); Five-fold cross-validation is adopted, with 80% of the samples used as the training set and 20% as the validation set during each fold of training.

[0215] 2.1 Age-conditional feature extraction;

[0216] The semantic embedding matrix E∈R output by the large language model T×d (Where T is the token sequence length and d is the embedding dimension) Input to the shared representation layer. First, global average pooling is performed on E along the time dimension to obtain a fixed-length semantic vector e∈R. d Then, the age codes a and e are concatenated and input into the fully connected network modulated by Age-FiLM to obtain the age-normalized shared feature vector z∈ (In this embodiment) =128).

[0217] 2.2 Multi-task forward computation;

[0218] Input z into the depression classification task head and the intervention prediction task head respectively, and calculate the depression classification loss L. cls (Cross-entropy loss) and intervention prediction loss L int (Binary cross-entropy loss). Simultaneously, z is input into the age regression task header to calculate the age regression loss L. age =MSE(age,age true This loss is used only during the training phase to force the shared representation to retain age information and prevent Age-FiLM from over-eliminating effective variations. The age regression task head works in conjunction with Age-FiLM, which is responsible for aligning the feature distributions of different ages, while the age regression task head prevents over-modulation and loss of necessary age information. Together, they improve the generalization ability across age groups.

[0219] 2.3 Cross-modal consistency loss;

[0220] To mitigate the impact of unreliable samples during the training phase using a pre-computed consistency metric, this embodiment employs a consistency-weighted loss without adding additional model parameters or a prediction head. Specifically, for each training sample, the confidence weight w for each time segment is read from the offline preprocessing. i Sample-level overall credibility .

[0221] Multiply the depression classification loss and intervention prediction loss by w respectively. sample Scaling:

[0222] ;

[0223] ;

[0224] Among them, L cls This represents the weighted depression classification loss, used for backpropagation to update model parameters;

[0225] This represents the raw cross-entropy loss (unweighted) calculated directly from the depression classification task head.

[0226] L int This represents the weighted predicted loss from intervention.

[0227] This represents the raw binary cross-entropy loss (unweighted) calculated directly from the intervention prediction task head.

[0228] W sampleThis represents the overall confidence level at the sample level, with a value range of [0,1], determined by the confidence weights w of all time series segments within that sample. i The average is obtained.

[0229] The advantages of this design are: (1) no additional parameters are required and the calculation is simple; (2) the gradient contribution of low consistency samples is directly reduced, avoiding unreliable data fitting of the model; (3) it is consistent with the goal of the conflict masking mechanism in the inference stage (both suppress the influence of inconsistent segments), forming a closed loop.

[0230] In an alternative implementation, a consistency prediction head can be added to directly predict consistency deviations using MSE loss, serving as an auxiliary task in multi-task learning. This embodiment prioritizes a weighted approach based on model simplicity and to avoid overfitting.

[0231] 2.4 Loss of developmental temporal continuity;

[0232] To force the model to learn a smooth developmental trajectory, for samples in the training set with follow-up data at multiple time points (e.g., baseline, 3 months, 6 months), the smoothness loss between shared feature vectors at adjacent time points is calculated:

[0233] ;

[0234] Where K is the number of sample pairs with follow-up data. This loss encourages the smoothing of feature representations of the same subject over time, reflecting the natural continuity of brain development, while suppressing aberrant jumps (e.g., sudden changes caused by emotional masquerading). In this embodiment, L dev This loss is calculated only on samples with at least two follow-up data points (which comprise approximately 30% of the training set). For single-assessment samples, this loss is automatically zeroed and does not participate in model optimization. This strategy ensures that the model can learn the smoothness of the developmental trajectory without rejecting single-assessment scenarios due to the mandatory requirement for longitudinal data.

[0235] 2.5 Dynamically weighted total loss;

[0236] The above losses are weighted and summed, and the weights of each loss term are automatically adjusted using a dynamic weighting method based on task uncertainty. Specifically, this embodiment introduces a learnable noise parameter σ for each loss term. i The total loss of the weighted summation is defined as:

[0237] ;

[0238] Among them, L cls The depression classification loss (cross-entropy) ranges from [0, +∞). The smaller the value, the more accurate the model is in identifying the depressive state.

[0239] L int The intervention prediction loss (binary cross-entropy) ranges from [0, +∞). The smaller the value, the more accurate the model's prediction of the intervention's effectiveness.

[0240] L dev The developmental temporal continuity loss (MSE) ranges from [0, +∞). The smaller the value, the smoother the shared feature vector of the same adolescent at adjacent follow-up time points, reflecting the natural continuity of the developmental trajectory.

[0241] L age is the age regression loss (MSE), with a value range of [0, +∞). The smaller the value, the more accurate the age information retained by the shared representation layer, and it is used to regularize Age-FiLM to prevent excessive removal of age information;

[0242] σ1, σ2, σ3, and σ4 are learnable noise parameters that are optimized along with the model parameters during training. The logarithmic term is a regularization term to prevent the noise parameters from growing indefinitely. This mechanism automatically assigns lower weights to tasks that are difficult to converge, eliminating the need for manual parameter tuning.

[0243] 2.6 Backpropagation and Early Termination

[0244] The parameters of the shared representation layer (including Age-FiLM parameters), depression classification task head, intervention prediction task head, and age regression task head (optional) are updated simultaneously using the backpropagation algorithm. An early stopping mechanism is employed during training: training terminates when the depression classification AUC on the validation set fails to improve for five consecutive rounds, and the model parameters with the highest validation set AUC are saved. Hyperparameters (such as learning rate, batch size, and weight coefficients) are determined through grid search.

[0245] 3. Verification of training effectiveness;

[0246] To verify the effectiveness of the above multi-task joint training process, this embodiment performs real-time evaluation of the validation set during the training process.

[0247] like Figure 3 The diagram shows the confusion matrix of the model trained in this embodiment on the validation set. The diagram shows that the model achieves an accuracy of 0.85 for healthy samples (label 0) and 0.86 for depressed samples (label 1). This indicates that after the training process including the constraints of "quality mask" and "cross-modal consistency loss," the model can converge effectively without exhibiting significant bias due to class imbalance.

[0248] like Figure 4The figure shows the receiver operating characteristic (ROC) curve of the model in this embodiment on the validation set. As shown, the area under the curve (AUC) of the model on the validation set reached 0.855 (±0.136). This result demonstrates that the training method used in this embodiment (especially the introduction of a cross-modal consistency loss term) can guide the model to learn more discriminative feature representations, thereby achieving excellent discrimination in the depression risk classification task.

[0249] This embodiment follows the data preprocessing process described in Embodiment 2, resulting in the following significant benefits during the training phase: The quality mask generated by preprocessing enables the model to perceive the data quality of each modality in each sample during training. Gradient contributions from low-quality segments are automatically suppressed during backpropagation, preventing noise contamination of model parameters. The cross-modal missing information completion step generates reasonable features and confidence levels for missing modalities in the training samples, ensuring the training set size is not reduced due to the absence of a particular modality. Simultaneously, the confidence level guides the model to utilize the completion information appropriately. The consistency detection step provides overall credibility for each segment, while the cross-modal consistency loss term during training further forces the model to learn consistent expressions between modalities. After training, the model, in conjunction with the conflict masking mechanism during the inference phase, effectively resists spoofing and false positives. Age-adaptive normalization aligns the feature distributions of subjects of different ages in the training samples, eliminating the need for the model to memorize normal ranges for different age groups. This allows the model to focus on learning pathological deviations, significantly improving the model's generalization performance across age groups.

[0250] In summary, this embodiment combines preprocessing with multi-task joint training and cross-modal consistency regularization to construct an end-to-end depression assessment and intervention prediction model that is insensitive to data quality, adaptable to age differences, and capable of identifying information conflicts. This model demonstrates high accuracy, robustness, and good interpretability in depression screening for adolescents (and adults).

[0251] Example 4

[0252] Finally, this application also proposes a computing device including a processor and a memory, wherein the memory stores a computer program and the processor executes instructions stored in the memory so that the computer device performs a depression assessment method based on heterogeneous multimodal physiological and behavioral data as described in the above embodiments.

[0253] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for assessing depression based on heterogeneous multimodal physiological and behavioral data, characterized in that, include: S1. Simultaneously acquire the subject's multimodal data and apply a unified timestamp and session identifier. The multimodal data includes at least text records, depression assessment scale data, audio data, video data, and brain function and physiological data. The brain function and physiological data includes at least functional magnetic resonance imaging data, functional near-infrared spectroscopy data, electroencephalogram data, eye movement data, and heart rate variability data. S2. The multimodal data are preprocessed respectively, and a quality mask corresponding to each modal data is generated; the quality mask is used to identify the validity, missing segments and low-quality segments of each modal data segment; Feature extraction is performed on the preprocessed modal data to obtain continuous features of each modal data; post-processing operation is performed on the continuous features; the continuous features after feature post-processing are discretized using a level mapping method based on the training data quantile threshold, and the continuous values ​​are mapped to multiple level intervals to generate a structured pseudo-text token sequence; wherein each token contains modality identifier, timestamp and feature level information; S3. The structured pseudo-text token sequence and the quality mask are uniformly embedded into the network, and after performing embedding encoding, temporal alignment, dynamic gating and cross-modal semantic fusion in sequence, a multimodal data vector is output. S4. Input the multimodal data vector and the scale cue token generated from the depression assessment scale data into the pre-trained large language model, use the scale cue token as a conditional cue to perform contextual semantic modeling, and output the semantic embedding matrix. S5. Input the semantic embedding matrix into the multi-task learning model, and output the depression assessment results and intervention effect prediction results.

2. The method according to claim 1, characterized in that, The feature post-processing operation in S2 includes an age-adaptive feature normalization step: The age information of the subjects is obtained and encoded to obtain an age feature vector; Based on the age feature vector, scaling and translation parameters of the affine transformation are generated through a fully connected network. Using the scaling and translation parameters, an affine transformation is performed on the continuous features to obtain age-normalized continuous features.

3. The method according to claim 1, characterized in that, The feature post-processing operation in S2 further includes a cross-modal missing data completion step: When any modality data is missing or its quality mask is marked as completely unavailable, a pre-trained cross-modal completion network is used to generate continuous features of the missing modality based on the continuous features of other available modalities, and the completion confidence is output. The completed continuous features are used together with the original continuous features for subsequent discretization, and the completion confidence is used as an additional quality mask in the weight adjustment in subsequent cross-modal semantic fusion.

4. The method according to claim 1, characterized in that, The feature post-processing operation in S2 further includes a cross-modal consistency detection step: The multimodal data is divided into several time-series segments based on the unified timestamp; Within each time segment, the consistency of at least two modality pairs is calculated: the first pair is the emotional consistency between audio data features and video data features, and the second pair is the physiological arousal consistency between EEG data features and heart rate variability data features; the emotional consistency between audio data features and video data features is obtained by mapping them to a shared emotional embedding space and calculating the cosine similarity; the physiological arousal consistency between EEG data features and heart rate variability data features is obtained by calculating the Pearson correlation coefficient between the power of a specific frequency band of EEG data and the time-frequency domain index of heart rate variability data. When the consistency of any pair of modalities is lower than a preset threshold, a collision mask for that modality is generated. For each mode pair, the quality masks of the two modes in the group are averaged to obtain the local quality score of the group. The conflict mask of the modality pair is fused with the local quality score to obtain the joint credibility of the modality pair in the time segment; The joint confidence scores of all modality pairs within the same time segment are weighted and combined to obtain the overall confidence score of the time segment; when the overall confidence score is lower than a first preset threshold, it is determined to be a low-confidence segment. The overall credibility and credibility determination results are associated with the pseudo-text tokens of the corresponding time segments to reduce the fusion weight of low-credibility segments in subsequent cross-modal semantic fusion.

5. The method according to claim 1, characterized in that, The embedding encoding in S3 includes: A token embedding layer is used to map each pseudo-text token to a fixed-dimensional token vector; A location embedding layer is used to add temporal location encoding to each token vector based on the unified timestamp and task anchor point; A paragraph embedding layer is used to add a dimension encoding that identifies the modality of origin for each token vector; The quality mask embedding layer is used to map the quality mask of each modality to a mask vector of the same dimension as the token vector through a linear transformation, and add or concatenate it with the token vector of the corresponding modality to obtain a token vector with quality information.

6. The method according to claim 1, characterized in that, The dynamic gating in S3 includes: Calculate the overall quality score for each modality based on the quality mask of each modality data; Completely block modal data whose overall quality score is lower than the second preset threshold; Output the masked modal data and the reasons for masking, which are used to generate a data quality report.

7. The method according to claim 6, characterized in that, The cross-modal semantic fusion in S3 includes: For modal data selected through dynamic gating, the fusion weight is determined in the attention mechanism based on the comprehensive quality score, wherein modal data with a higher comprehensive quality score receives a higher attention weight.

8. The method according to claim 1, characterized in that, Prior to S1, there is a training step for the multi-task learning model: S0A: Obtain historical multimodal sample data labeled with depression status and intervention effect labels, and construct a training dataset; S0B: Preprocess the training dataset to obtain the historical semantic embedding matrix; The historical semantic embedding matrix is ​​used to perform multi-task joint training on the initial multi-task learning model to obtain the multi-task learning model. The initial multi-task learning model is based on a multi-task learning architecture, including a shared representation layer, a depression classification task head for outputting depression risk level or binary classification results, and an intervention prediction task head for outputting intervention effectiveness probability or response hierarchical labels.

9. The method according to claim 8, characterized in that, The multi-task joint training in the S0B includes: The historical semantic embedding matrix is ​​input into the depression classification task head and the intervention prediction task head respectively, and the depression classification loss and the intervention prediction loss are calculated. Based on the historical semantic embedding matrix, cross-modal consistency loss is calculated, which includes audio-video emotional consistency loss and EEG-heart rate variability physiological arousal consistency loss. The depression classification loss, intervention prediction loss, and cross-modal consistency loss are weighted and summed, and the weights of each loss item are dynamically adjusted using an uncertainty weighting algorithm to obtain the total loss. Based on the total loss, the parameters of the depression classification task head, the intervention prediction task head, and the shared representation layer are simultaneously updated using the backpropagation algorithm.

10. The method according to claim 1, characterized in that, Following S5, the following is also included: At least one target modality token is selected from the semantic embedding matrix, and its discretization level is adjusted to a preset post-intervention target level to obtain the perturbed semantic embedding matrix. The perturbated semantic embedding matrix is ​​re-inputted into the multi-task learning model to recalculate the depression assessment result and calculate the amount of change between the original depression assessment result and the original depression assessment result. Based on the changes, the contribution of each modality feature to the depression assessment results is determined, and personalized intervention suggestion keywords are generated based on the contribution.

Images