Multi-modal depression recognition system based on MFE-CCAGNN model

Abstract

The invention belongs to the field of artificial intelligence, and provides a multi-modal depression recognition system based on an MFE-CCANNN model, which comprises a data acquisition unit, a data preprocessing unit and an MFE-CCANNN model unit. The data acquisition unit synchronously acquires multi-mode data such as videos, audios, texts and fNIRS when a subject performs the same interview task. The data preprocessing unit comprises a video preprocessing unit, an audio preprocessing unit, a text preprocessing unit and an fNIRS preprocessing unit. The MFE-CCARNN model unit comprises a video, audio, text and fNIRS neural signal feature extraction module, a multi-modal feature fusion module and a classification module, and depression recognition and classification result output are achieved. The system supports four-level depression degree discrimination, is high in recognition precision, portable in deployment, high in interpretability and the like, and is suitable for psychological health screening and clinical auxiliary evaluation scenes.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, and specifically relates to a multimodal depression recognition system based on the MFE-CCAGNN model. Background Technology

[0002] Depression is a common mental disorder, clinically manifested as symptoms such as depressed mood and loss of interest, and in severe cases, it can even lead to self-harm or suicide. Current clinical diagnostic methods primarily rely on professional interviews with physicians and assessments using depression scales such as the Hamilton Depression Rating Scale (HAMD), the Beck Self-Rating Depression Scale (BDI), the Self-Rating Depression Scale (SDS), and the Patient Health Questionnaire-9 (PHQ-9). However, these methods are highly dependent on the physician's expertise and the patient's cooperation, and suffer from limitations such as strong subjectivity, resulting in limitations in diagnostic efficiency and consistency. Furthermore, due to individual cognitive biases, mood swings, or societal expectations, some patients may not accurately reflect their true condition. Some patients may even underestimate or exaggerate their symptoms due to shame or other psychological reasons, leading to misdiagnosis or missed diagnosis during clinical assessment. Therefore, given limited medical resources, there is an urgent need for an automated tool to assist in the initial screening and identification of depression in clinical practice.

[0003] With the development of machine learning technology, existing methods have attempted to detect depression based on single modalities such as facial expressions and speech. For example, convolutional neural networks are used to analyze facial expressions, or support vector machines and decision trees are used to mine features such as pitch and volume in speech. However, these single-modal methods are often susceptible to data occlusion, environmental noise, and contextual differences, resulting in unstable detection performance, insufficient generalization ability and personalized adaptability, and difficulty in achieving satisfactory results in complex real-world scenarios.

[0004] To improve accuracy and robustness, some studies have introduced multimodal data from external behaviors (such as video, audio, and text) for multimodal fusion, as this data is relatively easy to obtain. Methodologically, joint modeling using convolutional networks, Transformers, or gated recurrent structures offers a certain improvement in performance compared to single-modal recognition.

[0005] Meanwhile, some studies use physiological signals that can reflect the state of neural response, such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS), to detect depression, and their performance is relatively outstanding.

[0006] However, the use of multimodal fusion of external behaviors overlooks the correspondence between external behaviors and an individual's internal neural state. For example, the same "smile" might be a social response in a greeting context, represent positive emotions in a pleasant context, but imply suppressed emotions in a family conflict dialogue. Without alignment with neural responses, external cues are prone to contextual ambiguity. Furthermore, relying solely on physiological data of neural responses has significant limitations. First, data acquisition is highly dependent on equipment and environment; for example, fMRI is costly and sensitive to motion artifacts, and EEG is easily affected by noise and electrode contact, making it difficult to promote in large-scale population screening. Second, physiological indicators are affected by individual differences, current state, and cross-device differences, resulting in limited consistency and comparability across populations and scenarios. Third, using neural signals alone, without alignment with specific external behaviors (and context), makes it difficult to determine the true emotional meaning corresponding to the same physiological change, leading to insufficient interpretability and a high risk of false positives or false negatives.

[0007] In summary, existing technologies either focus solely on external behavior while neglecting internal neural states, or rely solely on neurophysiology while detaching from the real-world interaction context, making it difficult to simultaneously ensure stability, interpretability, and deployability. Therefore, there is an urgent need for a method that, within a unified interview task paradigm, can synchronously map external behavioral modalities to internal neural responses. This would eliminate the lack of internal neural modal support for external behavioral modalities and resolve contextual ambiguities in internal neural modalities, thereby improving the reliability of depression identification. Summary of the Invention

[0008] The purpose of this invention is to overcome the limitations of existing depression recognition systems that rely solely on external behavior or neurophysiological signals, and to propose a multimodal depression recognition system based on MFE-CCAGNN (Multimodal Feature Extraction and Cross-Modal Co-Attention with Graph Neural Networks). During the training phase, the system simultaneously acquires external behavioral modalities (video, audio, and text) and functional near-infrared spectroscopy (fNIRS) neural responses, enabling explicit alignment between external behavior and internal neural responses. During deployment, it can automatically generate neural proxy features for discrimination based solely on external behavior without relying on neural modalities, thereby maintaining interpretability while reducing device dependence and improving the stability and scalability of the screening process.

[0009] In this invention's system, external behavioral modalities (video, audio, text) are explicitly aligned with fNIRS neural responses to gain a deeper understanding of the internal-external mapping relationship. This allows the system to automatically generate neural proxy features corresponding to neural representations based solely on external behaviors for discrimination in future scenarios such as large-scale depression identification where neural modalities are missing. This maintains interpretability while reducing device dependence and improving the stability and scalability of screening. Considering the requirements of interview scenarios for natural interaction and ease of deployment, fNIRS possesses engineering advantages such as portability, tolerance to mild head movements, and ease of deployment alongside cameras and audio pickup devices, making it suitable as a source of teacher anchor points during system training. Based on this, this invention proposes a specific technical solution for a depression identification system.

[0010] Technical solution of the present invention: A multimodal depression recognition system based on the MFE-CCAGNN model includes a data acquisition unit, a data preprocessing unit, and an MFE-CCAGNN model unit, wherein: The data acquisition unit is used to simultaneously collect video modal data, audio modal data, text modal data, and fNIRS modal data of the subjects in a standardized interview task scenario during the training phase, and to collect only external behavioral modal data in the same task scenario during the deployment phase. The external behavioral modal includes video, audio, and text. The data preprocessing unit includes a video preprocessing unit, an audio preprocessing unit, a text preprocessing unit, and an fNIRS preprocessing unit; The MFE-CCAGNN model unit includes four feature extraction modules: video feature extraction module, audio feature extraction module, text feature extraction module, and fNIRS data feature extraction module, as well as a multimodal feature fusion module and a classifier module; The data acquisition unit provides time-aligned raw data support for the subsequent data preprocessing unit; the inputs of each feature extraction module of the MFE-CCAGNN model unit are connected to the outputs of the corresponding preprocessing modules of the data preprocessing unit to extract features from various types of data output by the data preprocessing unit; the multimodal feature fusion module of the MFE-CCAGNN model unit completes explicit alignment, sharing, and unique embedding decoupling and neural proxy feature generation during the training phase, and generates neural proxy features based solely on external behavior without using neural modalities during the deployment phase, and fuses them with external behavior features, and finally the depression recognition result is output by the classifier module.

[0011] The design principles of the above system technical solution are as follows: By designing a unified data collection unit, four modalities (video, audio, text, and fNIRS) were simultaneously collected under a standardized interview task, ensuring temporal consistency and correspondence with psychological states. Based on this synchronous collection mechanism during the training period, external behavior and internal neural responses were placed in the same temporal context, overcoming the problems of incomplete expression and insufficient robustness of single-modal and multimodal methods that rely solely on external behavior in representing depression-related information.

[0012] Meanwhile, the accompanying four data preprocessing units—video, audio, text, and fNIRS—complete the structured feature extraction for the video modality, the speech denoising and standardization for the audio modality, the language text cleaning for the text modality, and the neural signal correction processing for the fNIRS modality, significantly improving the quality and usability of multimodal data.

[0013] Finally, the MFE-CCAGNN model unit includes video, audio, text, and fNIRS neural signal feature extraction modules, a multimodal feature fusion module, and a classifier module. The video feature extraction module employs an emotion-gating mechanism—the Transformer. The audio feature extraction module uses multi-channel convolution for feature extraction. The text feature extraction module uses an N-Transformer for feature extraction. The fNIRS feature extraction module uses multi-scale convolution for feature extraction. The multimodal feature fusion module uses a shared-unique embedding representation decoupling fusion mechanism based on graph convolutional neural networks and an attention fusion strategy to achieve joint representation modeling of behavioral, linguistic, and neural multidimensional pathways. Finally, the classifier module implements depression recognition and outputs classification results.

[0014] The beneficial effects of this invention are: The system of this invention supports four levels of depression discrimination, with high recognition accuracy, easy deployment, and strong interpretability, making it suitable for mental health screening and clinical auxiliary assessment scenarios. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall system design of the present invention; Figure 2 This is a schematic diagram of the data acquisition process; Figure 3 A schematic diagram illustrating the implementation of the video feature extraction module for the video preprocessing unit and the MFE-CCAGNN model unit; Figure 4 A schematic diagram illustrating the implementation of the audio feature extraction module for the audio preprocessing unit and the MFE-CCAGNN model unit; Figure 5 This diagram illustrates the implementation of the text preprocessing unit and the text feature extraction module of the MFE-CCAGNN model unit. Figure 6A schematic diagram illustrating the implementation of the fNIRS feature extraction module for the fNIRS preprocessing unit and the MFE-CCAGNN model unit; Figure 7 A schematic diagram illustrating the implementation of the multimodal feature fusion module for the MFE-CCAGNN model unit; Figure 8 The prediction results of the depression recognition system under four stimulus tasks; Figure 9 Comparison of classification performance of different modalities under four stimulus types for the system input during the deployment phase of the depression recognition system; Figure 10 A performance comparison between the MFE-CCAGNN model unit and current mainstream deep learning models during the deployment phase of a depression recognition system. Detailed Implementation

[0016] The technical solutions provided in this application will be further described below with reference to specific embodiments and accompanying drawings. The advantages and features of this application will become clearer from the following description.

[0017] like Figure 1 As shown: A multimodal depression recognition system based on the MFE-CCAGNN model includes a data acquisition unit, a data preprocessing unit, and an MFE-CCAGNN model unit, wherein: The data acquisition unit is used to simultaneously collect video modal data, audio modal data, text modal data, and fNIRS modal data of subjects in a standardized interview task scenario during the training phase, and to collect only external behavioral modal data (video, audio, and text) in the same task scenario during the deployment phase.

[0018] The data preprocessing unit includes a video preprocessing unit, an audio preprocessing unit, a text preprocessing unit, and an fNIRS preprocessing unit; The MFE-CCAGNN model unit mainly includes four feature extraction modules: video feature extraction module, audio feature extraction module, text feature extraction module, and fNIRS data feature extraction module, as well as a multimodal feature fusion module and a classifier module; The data acquisition unit provides time-aligned raw data support for subsequent data preprocessing units. The inputs of each feature extraction module in the MFE-CCAGNN model unit are connected to the outputs of the corresponding preprocessing modules in the data preprocessing unit, extracting features from various types of data output by the data preprocessing unit. During the training phase, the multimodal feature fusion module of the MFE-CCAGNN model unit completes explicit alignment, sharing, and unique embedding decoupling, as well as neural proxy feature generation. During the deployment phase, it can generate neural proxy features based solely on external behavior without using neural modalities and fuse them with external behavioral features. Finally, the classifier module outputs the depression recognition result.

[0019] like Figure 2 As shown: The data acquisition unit involves existing equipment, and the hardware itself is not a technical innovation of this invention.

[0020] Specifically, the video modal data in the data acquisition unit mainly refers to facial video data of the subjects during the interview task. The facial video data can be further used to extract biorepresentational data such as the location of facial key points, head posture, and eye movement behavior during the interview task (the extraction process will be completed in the video preprocessing unit of the data preprocessing unit). Individuals with depression typically exhibit a lack of facial expression, reduced positive expressions, and increased negative expressions. Common facial action unit (AU) combinations may show pattern changes (e.g., a significant decrease in smiling frequency). In terms of facial geometry, tendencies such as drooping corners of the mouth, eyelid laxity, and decreased facial muscle tension can be quantified through facial key point trajectories. Regarding head posture, depressed individuals may often exhibit behavioral characteristics such as looking down, avoiding eye contact, and reduced head movement amplitude; relevant information can be captured by posture angles (Pitch / Yaw / Roll). Eye movement behavior also shows typical differences, manifesting as shortened fixation time, eye avoidance, and blank stares, with relatively slow changes in eye movement direction.

[0021] Specifically, the audio modal data in the data acquisition unit mainly refers to the voices of subjects answering questions during interview tasks. Common characteristics of depressed patients during verbal communication include slower speech rate, flat tone, decreased vocal energy, and increased pauses and delays. These non-semantic acoustic features can be used to reflect their psychological state without relying on semantic understanding. The dynamic temporal continuity of audio data can capture the patient's emotional changes in real time during communication. For example, depressed patients typically have a more monotonous tone, lacking high and low pitch, a significantly slower speech rate, longer pauses, and a sustained low-energy vocal output, even including speech interruptions and trembling or unstable voices. These differences enable audio modalities to represent depression-related states.

[0022] Specifically, the text modal data in the data acquisition unit mainly refers to the semantic content of the subjects' answers to questions during interview tasks. Individuals with depression may exhibit patterns such as an increased proportion of negative emotional words, increased use of first-person pronouns, reduced active speech, and relatively impoverished emotional expression at the linguistic content level. These changes in semantic and psychological characteristics can be modeled and quantified through text modality analysis, such as changes in the frequency of use of words like "disappointment," "pain," and "helplessness," changes in the proportion of first-person pronouns, and sentence structure and fluency indicators, to reflect trends such as cognitive negativity, emotional avoidance, and simplified expression. In this invention, the text modal data from the data acquisition unit is used for identification and not for the clinical differential diagnosis between different mental disorders.

[0023] Specifically, the fNIRS modal data in the data acquisition unit mainly refers to the changes in infrared light intensity in the cerebral cortex regions of subjects collected by functional near-infrared spectroscopy equipment during interview tasks, cognitive stimulation, or emotion induction. At the neurophysiological level, depressed patients generally exhibit reduced neural activation levels in the prefrontal cortex (such as the dorsolateral prefrontal cortex DFLPFC and medial prefrontal cortex mPFC) during tasks such as verbal expression, emotional arousal, or memory recall. In terms of neural representation, depressive states may be accompanied by reduced activation, low synchronicity of functional networks, and cortical deactivation patterns, demonstrating insufficient response to external task stimuli and decreased neural processing efficiency. Therefore, fNIRS can be used to measure the task-oriented state of depressed individuals, allowing for timely acquisition of abnormal response patterns in their cerebral cortex. Under task-induced conditions, the task-increase in hemoglobin (HbO) (ΔHbO) in fNIRS signals of depressed individuals may be lower than that of healthy controls, and the peak time may be delayed. In this invention, the fNIRS modality serves as a teacher anchor point during the training phase, used for explicit alignment with the external behavioral modality to learn the mapping relationship between external behavior and internal neural states. During the deployment phase, fNIRS is no longer collected; instead, neural proxy features generated from external behavior participate in the discrimination.

[0024] The four modalities—video, audio, text, and fNIRS—are complementary in terms of information dimensions and data sources, representing two key feature sources: external behavioral performance (video, speech, and text) and internal neurophysiological state, respectively. The combined use of these four modalities supports synchronous alignment and mapping learning during the training phase, and maintains an approximate representation of the "behavior-neuron" relationship during the deployment phase through neural proxy features generated from external behavior. This balances interpretability and deployability in the recognition task.

[0025] The algorithm design of each module is described in detail below.

[0026] like Figure 3 As shown: The video preprocessing unit of the data preprocessing unit is used to transform the raw video modal data into structured, modelable low-level structured descriptors, and finally output video pre-processing data (VPRE). This preprocessing process is applicable in both the training and deployment phases of the system.

[0027] Specifically, the video preprocessing unit includes a face detection module, a video frame extraction module, and a low-level descriptor extraction module, which are connected in sequence: (1) The face detection module accurately locates the subject's facial area from the input raw video data, removes background and non-facial interference information, and ensures that subsequent processing focuses on the effective area; (2) Video frame capture module, used to capture frames that can remove redundant frames and unify the face orientation, reduce noise interference caused by pose differences, and perform frame-level sampling at 0.1-second intervals; (3) The low-level descriptor extraction module is built using the open-source tool OpenFace to extract data such as facial key point locations, head posture, and eye movement behavior of the subjects during the interview task (these data can be used as biometrics of facial expressions in depression identification). The data output by the low-level descriptor module includes facial expression action unit data, including: Facial motion unit (AUs) intensity: intensity values ​​of 17 motion units; Facial key points: 68 two-dimensional coordinate points, totaling 136 dimensions; Head pose: X / Y / Z 3D position and Pitch / Yaw / Roll rotation angles, totaling 6 dimensions; Eye movement direction: a three-dimensional vector of the left and right eye gaze directions, totaling 6 dimensions.

[0028] The final output VPRE of the video preprocessing unit is 165 dimensions.

[0029] like Figure 3 As shown: The video feature extraction module of the MFE-CCAGNN model unit takes as input the video coding features (VPRE) output by the video preprocessing unit, and its output is... For ease of subsequent description, the frame sequence of VPRE will be denoted as... Each frame .

[0030] Typical facial behavioral changes in patients with depression often exhibit characteristics of being "non-drastic," "low-activity," and "slowly evolving," such as decreased facial muscle tone, reduced smiling frequency, eye avoidance, and shortened fixation time. These behavioral features are expressed at low frequency, short duration, and unevenly distributed at the frame level, causing conventional models (such as traditional Transformer or Bi-LSTM) to be easily diluted by a large number of neutral frames during the encoding process, thus failing to accurately focus on emotion-related changes.

[0031] To resolve the conflict between "sparse emotion frames" and "inter-frame information interference," this module focuses on micro-expression fluctuations and abnormal facial movement unit expressions closely related to depressive states. The video feature extraction module constructs an emotion activity gating-Transformer structure, which includes an emotion scoring calculation block, an emotion activity gating weighting block, and a Transformer encoder. The Transformer encoder structure has two stacked layers, each containing four multi-head attention heads (32 dimensions each), resulting in a total input dimension of 165. The feed-forward network (FNN) has a dimension of 512, and both layers employ residual connections and LayerNorm normalization mechanisms. This module is used in both the training and deployment phases.

[0032] Its working process is as follows: (1) Input features and emotion scores are calculated using the emotion score calculation block. Each time frame in the VPRE data To represent the structured video behavioral features of this frame, the emotion score for each frame is first calculated. And compressed to the (0,1) interval, in This represents the Sigmoid function.

[0033] (2) Perform emotional activity gating weighting operation through emotional activity gating weighting block. A gating structure is used to fuse frame features guided by sentiment weights with the original features. , in , , It's based on emotions. and These are the frame feature representations before and after weighting, respectively, and ReLU(·) is a non-linear activation function.

[0034] (3) Construct the Transformer encoder. (3.1) First, sequence construction and linear mapping are performed: all frames are constructed into a new video sequence. And use linear mapping to perform embedding dimension transformation to adapt to the Transformer structure, while keeping the input dimension at 165.

[0035] (3.2) Secondly, multi-head attention modeling of inter-frame dynamic association is performed: a stacked multi-head self-attention mechanism is used to perform context modeling of the emotion-weighted frame sequence. , in It is a stacked calculation. , It is the output mapping matrix, and . The above Q, K, and V are the query, key, and value vector sequences, respectively. It is a multi-class classification function. In this embodiment, the number of attention heads... The dimension of a single head is The dimensions after splicing are .

[0036] (3.3) Finally, nonlinear transformation and residual connection are performed: the output sequence is transformed through a feedforward network and residual connection to obtain the output of the video feature extraction module. , in Indicates a feedforward network. It is layer normalization. This represents the output of the video feature extraction module.

[0037] like Figure 4 As shown: The audio preprocessing unit (hereinafter referred to as the audio preprocessing unit) of the data preprocessing unit processes the raw audio speech data and finally outputs audio pre-processing data (APRE). This preprocessing process is applicable in both the training and deployment phases.

[0038] The audio preprocessing unit includes a filtering and noise reduction module, an amplitude normalization module, and a silence data removal module, which are connected in sequence as follows: (1) Filtering and noise reduction module. This module uses a Butterworth bandpass filter to perform frequency domain filtering on the original audio data, filtering out environmental interference signals that are too high or too low, retaining the human voice frequency band in the range of [80Hz, 8000Hz], and filtering out low-frequency noise, current interference and equipment noise.

[0039] (2) Amplitude normalization module. The function of this module is to perform amplitude normalization processing on the denoised audio data to eliminate the influence of volume differences between different individuals or recording environments. Specifically, the amplitude normalization module of the audio preprocessing unit can resample the original acquired speech to 16kHz and perform amplitude normalization to standardize its amplitude range to [-1,1].

[0040] (3) Silence Data Removal Module. This module removes silence segments from the normalized audio data to eliminate invalid pauses and background noise interference, preserving valid speech segments. Specifically, the silence data removal module of the audio preprocessing unit removes silence segments using a frame energy determination method, as shown below: , in, For frame energy, It is the input voice data. Represents a speech segment, This represents the total number of speech segments. If the energy of a given frame satisfies... ,in If the threshold for silence is set, the frame is identified as an invalid silent frame and discarded.

[0041] (4) Mel power spectrum calculation module. This module calculates the Mel-spectrogram of the audio data after removing silence. The Mel-spectrogram reflects changes in rhythm, pitch, and intensity in the speech signal, and reflects abnormal speech patterns commonly seen in individuals with depression, such as slowed speech rate, flat tone, and decreased energy. Specifically, this module uses short-time Fourier transform to calculate the power spectrum and maps it to the perceptual frequency space using a Mel filter bank to generate a 128-dimensional Mel frequency energy spectrum. The calculation formula is as follows: . in It refers to the audio frequency of the voice data. Depend on It was converted from; It corresponds to the Mel frequency, which is the APRE output by the audio preprocessing unit.

[0042] like Figure 4 As shown: The audio feature extraction module (hereinafter referred to as the audio feature extraction module) of the MFE-CCAGNN model unit takes the APRE output by the audio preprocessing unit as input and outputs the audio encoded features. .

[0043] Depressed patients often exhibit nonlinear changes in speech during natural speech communication, such as slowed speech rate, flat tone, decreased intensity, and weakened frequency fluctuations. These characteristics are often reflected in the Mel spectrum obtained by the audio preprocessing unit as sliding frequency variations, rhythmic imbalances, and decreased energy density. To capture these multi-scale speech features, the audio feature extraction module uses convolutional kernels of different scales in parallel to extract local details, short-term rhythm, and frequency band fluctuation information, splicing them together to form a fused spectral domain representation. Simultaneously, a channel residual structure is used to maintain the consistency of the spectral structure through the fusion of the original input and the multi-scale convolutional outputs, improving the robustness against perturbations in the deep feature extraction stage.

[0044] In the audio feature extraction module, the multi-scale convolutional structure contains three one-dimensional convolutional kernels with sizes of 1×1, 1×3, and 1×5, respectively. Each kernel has 64 channels, and the total number of output channels after concatenation is 192. The convolutional layer is followed by BatchNormalization and ReLU activation. To ensure temporal alignment with the original spectrogram, the convolution uses a stride of 1 and same padding to maintain a constant temporal length.

[0045] Its working process is as follows: (1) First, for the input APRE data, 1×1, 1×3, and 1×5 convolutional kernels are constructed to extract local details, short-term rhythm, and sliding frequency band changes, respectively. The outputs of the three are concatenated to form a feature fusion representation. , in The input APRE represents the audio data. This represents a one-dimensional convolution along the time dimension, with each branch having 64 output channels. Concatenate according to channel dimension, with k being 1, 3, or 5.

[0046] (2) Channel normalization and activation. For fusion features... Performing BatchNorm and ReLU operations improves the network's nonlinear modeling ability and numerical stability.

[0047] . in, The activation feature has the same duration and 192 channels.

[0048] (3) Channel residual fusion. Features after activation With the original input After addition, the final output of the audio feature extraction module is: .

[0049] like Figure 5 As shown: The text preprocessing unit (hereinafter referred to as the text preprocessing unit) of the data preprocessing unit preprocesses the acquired raw text data and finally outputs text preprocessing data (TPRE).

[0050] The text preprocessing unit is a data cleaning and standardization module, which is a Python script file. Running in a Python 3.8 environment, this file can clean and standardize text data. The script's processing flow includes removing interjections (such as "um" and "ah"), non-linguistic noise markers (such as prompts from medical staff), meaningless repetitions (such as converting "I-I-I think" to "I feel"), and correcting abnormal characters or sentences to eliminate redundant and interfering information and improve the purity and stability of semantic modeling. Language phenomena such as misspellings, sentence interruptions, and repeated breaks are preserved without automatic semantic rewriting or polishing.

[0051] like Figure 5 As shown: The text feature extraction module (hereinafter referred to as the text feature extraction module) of the MFE-CCAGNN model unit takes the TPRE output by the text preprocessing unit as input and outputs the text encoded features as output. .

[0052] In natural communication, patients with depression often exhibit characteristics such as sparse semantic structure, disordered language expression, redundant interjections, low density of emotional words, and high rate of grammatical errors. They are also prone to linguistic noise phenomena such as pauses, interfering words, and structural omissions.

[0053] To address this, the text data feature extraction module constructs an N-Transformer structure, which includes multiple Transformer blocks and employs a stacked multi-head attention structure to model the contextual relationships between words. Emotional region attention is introduced into the embedding layer to enhance the position of negative emotion words and first-person pronouns, strengthening the model's ability to perceive depressive tendencies and output stable features. In the text feature extraction module, the maximum length of the input text sequence is 128. First, the word vectors are projected onto a dimension of [dimension not specified in the original text]. d In a space of 256, a text embedding matrix is ​​formed; the subsequent input is fed into an 8-layer Transformer structure, each Transformer layer contains 8 attention heads, each attention head has a dimension of 32, and the intermediate dimension of the feedforward network is 512.

[0054] Its working process: (1) Embedding layer and sequence representation. The sequence representation of the Text Preprocessing Unit (TPRE) output is as follows: , , in It can be represented as the first i The maximum sequence length of the words is d =128. The word vectors are mapped to an embedding space of dimension d through an embedding layer, resulting in the embedding matrix. , is represented as: in It is an embedding layer that can achieve the fusion of word or subword vectorization and positional encoding.

[0055] (2) Multi-head self-attention and emotion region attention masking. Embedding matrix The input is fed into a multi-layered stacked Transformer module (identified as 8 layers in this invention), each layer containing a multi-head self-attention module and a feedforward network module (intermediate layer dimension is 512). The standard multi-head self-attention calculation is as follows: . Q, K, and V are obtained by linear mapping from the input embedding matrix. The dimension is for a single-headed key vector. For global attention, only region attention needs to be considered; therefore, a mask is used to construct the region attention based on grammatical error tags. in This represents the original attention distribution. This represents the attention area mask constructed from grammatical error tags (1 indicates preservation, 0 indicates masking).

[0056] The output for each attention is: Finally, we obtain the context feature representation matrix after Transformer encoding. (3) [CLS] Convergence and Dual-Channel Output. From the context feature matrix Extract the [CLS] flag bit vector (Representing a global semantic summary), using dual-channel output to compute contextual features for a sentiment semantic representation. .

[0057] Language consistency score of context features Finally, the text features are merged. , in This represents the output of the text feature extraction module. This represents the sigmoid function.

[0058] like Figure 6 As shown: The fNIRS preprocessing unit of the data preprocessing unit (hereinafter referred to as the fNIRS preprocessing unit) processes the fNIRS signals collected from the subjects during the interview task or cognitive stimulus process, and finally outputs fNIRS preprocessing data (NPRE).

[0059] Specifically, the fNIRS preprocessing unit includes an HbO concentration estimation module, a bandpass filtering and noise reduction module, and an HbO signal baseline correction module, which are connected in sequence as follows: (1) HbO concentration estimation module. This module first performs a logarithmic transformation on the fNIRS light intensity data to obtain the change in optical density, and then uses the modified Bell-Lambert law to estimate the change in hemoglobin concentration. .

[0060] (2) Bandpass filtering and noise reduction module. The function of this module is to reduce the noise of the original signal. Frequency band limiting is implemented to filter out non-cognitive components. Specifically, the bandpass filtering and noise reduction module mainly uses an FIR bandpass filter to limit the frequency range to 0.01–0.1Hz to retain low-frequency components relevant to the task and suppress low-frequency drift and high-frequency noise.

[0061] (3) HbO signal baseline correction module. This module can perform baseline correction on each channel. Baseline correction is performed by shifting the signal to remove baseline deviation during the resting period. The specific calculation method is as follows. , in, This represents the task increment of hemoglobin concentration in a certain channel at time t before baseline correction. ), Represented as baseline corrected , This indicates the duration of the resting period, used to estimate the baseline mean. It is an integral variable.

[0062] like Figure 6 As shown: The fNIRS feature extraction module (hereinafter referred to as the fNIRS feature extraction module) of the MFE-CCAGNN model unit takes the NPRE output by the fNIRS preprocessing unit as input and outputs the fNIRS encoded features. .

[0063] In individuals with depression, activation levels in areas such as the prefrontal cortex and cingulate cortex may be reduced during emotional processing and task execution, and functional connectivity between brain regions exhibits desynchronization and heterogeneous activation patterns. In fNIRS data, this is reflected in weakened HbO signal correlation and decreased connectivity stability between channels. To address this, the fNIRS feature extraction module first calculates the Pearson correlation matrix between different channels to describe neural coordination patterns under task conditions. Subsequently, to address issues such as high-dimensional redundancy, low-frequency drift, and complex connectivity patterns caused by multi-channel acquisition, two-dimensional convolution and multi-path pooling (including max pooling, average pooling, and attention pooling) are employed. This module is used for mapping learning and explicit alignment during the training phase, but fNIRS is not invoked during the deployment phase.

[0064] The fNIRS feature extraction module of the MFE-CCAGNN model unit includes a Pearson correlation matrix calculation module, a two-dimensional convolutional block, and a multi-path pooling block. Specifically, the input is a 53×53 correlation matrix (the fNIRS device output data format is 53 channels), the convolutional kernel size is 3×3, the number of channels is 64, and the multi-path pooling block uses three pooling paths: max pooling, average pooling, and attention pooling.

[0065] Its workflow is as follows: (1) The Pearson correlation coefficient of HbO signals between all channels was calculated using the Pearson correlation matrix calculation module. A 53×53 connectivity matrix was output, which represents the co-activation patterns of different brain regions during task execution and characterizes the neuromodulation features of an individual in a depressive state. , in and This is a channel i and j The HbO signal.

[0066] (2) Use convolution kernels to extract local neural connectivity patterns. (3) The convolutional features are subjected to three parallel pooling methods: max pooling to capture high-intensity co-activation regions, average pooling to characterize the overall connectivity stability, and attention pooling to improve the model's ability to focus on key connectivity pathways. Among these, max pooling features... F fnirs-max Average pooling characteristics F fnirs-mean Attention pooling features F fnirs-attn The calculations are as follows: , , , in It is the importance weight of the attention pooling learning region. The three types of pooling features are concatenated along the channel dimension to form a multi-view neural connectivity graph representation. . like Figure 7 As shown: The multimodal feature fusion module takes as input the outputs of the four feature extraction modules mentioned above. , , , During the training phase, deep embedding mining and explicit alignment are performed on the four types of features, and optimization is carried out using cross-modal contrastive and latent space reconstruction with joint orthogonal constraints; during the deployment phase (missing neural modalities), only... , , As input, generate neural agent features corresponding to NFS and fuse them with external behavioral features, outputting weighted fused features. .

[0067] The output features of the four feature extraction modules represent the salient features of each modality, but the intensity, duration, and manifestation of these features often differ. Using only concatenation or uniform embedding strategies can easily lead to intermodal interference, suppressing key signals or introducing redundant information. On the other hand, multimodal signals within the same task context simultaneously contain common representations shared with other modalities and unique representations manifested only in the current modality. For example, in the question-and-answer session "Do you feel happy?", the video modality includes unique embedding representations such as micro-expressions, as well as common emotional responses consistent with speech and text. Therefore, the fusion module employs a shared-unique embedding representation decoupling fusion mechanism, ensuring the coordination of common information while preserving the differentiated cues of each modality.

[0068] To address the complex characteristics of multimodal data in depression recognition tasks, which exhibit both commonalities and differences in expression across modalities, a multimodal feature fusion module is constructed based on a shared-unique embedding representation decoupling fusion mechanism to achieve robust fusion of multimodal features. This shared-unique embedding representation decoupling fusion mechanism involves applying orthogonal constraints to shared and unique embeddings within the same modality, combining embedding representation reconstruction and shared space alignment using fNIRS as teacher anchors, and generating neural proxy feature reconstruction from the inside out, thereby achieving robust cross-modal fusion.

[0069] Its workflow is as follows: (1) Dimensional uniform representation.

[0070] To perform display alignment and subsequent fusion within the shared space, the four modal features VFS, AFS, TFS, and NFS are first uniformly mapped to the same dimension. This process uses a two-layer MLP to perform dimension alignment on each modality, resulting in an equal-dimensional representation: , , in It is an element-wise nonlinear activation function. and These are the weight matrix and bias vector of the first-layer MLP, respectively. and These are the weight matrix and bias vector of the second-layer MLP, respectively. The final output is the modality. Node feature matrix Correspondingly, , , , These are the same-dimensional representations of video, audio, text, and fNIRS modalities.

[0071] (2) Embedded representation extraction (2-1) Extraction of shared embedding representation Will In a GCN shared encoder with shared input parameters, the output is a shared embedding representation of four modalities. , , and During the training phase, all four modalities are input, and during the deployment phase, the external behavioral modality is used. participate.

[0072] Shared graph construction: for each modality The image is recorded as follows The set of nodes is , Representing modes m The The input feature representation of each node. Edge set. The K-Nearest Neighbors (KNN) algorithm is used to measure the Euclidean distance of each node in the semantic space, and each node is connected to its K nearest neighbors in the feature space by an edge. in This represents the distance between two objects; the smaller the value, the closer they are. (In the adjacency matrix...) Add self-connects and normalize the adjacency matrix. , , in for The degree matrix, It is the identity matrix. For each mode... m Using the same set of shared parameters By performing a graph convolution mapping function, the shared embedding representation is obtained as follows: , in It is an element-wise nonlinear activation function.

[0073] (2-2) Unique Embedding Representation Extraction Will A unique GCN encoder with non-shared input parameters and a unique embedded representation of the output modality. , , and During the training phase, all four modalities are input; during the deployment phase, the external behavioral modality is used. participate.

[0074] Unique graph construction: for each modality , record the unique picture ,in, A unique graph network structure for each mode m, Representing modes The set of nodes below, This represents the edge connections between nodes.

[0075] For unique images Unique adjacency matrix Using an independent KNN strategy, the nearest neighbor structure is computed separately for each mode (modes do not share structures with each other). The adjacency matrix and its normalized form are as follows: , , in This is a node degree matrix, with diagonal elements representing the number of connections for each node. For each mode... The parameters of the unique encoder are not shared; they are used to capture video, semantic, structural, and physiological features specific to each modality. , in The GCN weight matrix is ​​learned separately for each modality in a unique encoder. It is a non-linear activation function.

[0076] (3) Shared-unique embedded representation decoupling and fusion mechanism The goal of decoupling and fusing shared-unique embedded representations is to construct a joint optimization loss objective around three objectives: orthogonal constraints, embedded representation reconstruction, and shared embedded representation alignment.

[0077] (3-1) Orthogonal constraints of shared and unique embedding representations Shared Embedded Representation Representing cross-modal general information, modality-unique embedding representation Capture modality-specific differences. Since shared and unique encoders may extract redundant or overlapping information, representation confusion can easily occur. Therefore, an approximate orthogonal constraint is applied to the shared and unique representations of the same modality to make them as linearly independent as possible. To achieve the above constraints, an orthogonal constraint loss function is introduced, requiring the calculation of the orthogonal constraint loss for each mode individually. For the embedding representations of all four modes mentioned above, the orthogonal constraint loss is, . in It is the Frobenius norm. This represents the sum of squares of all elements. This loss will be incorporated into the model. The above loss is included in the joint optimization objective during the training phase, with the weighting coefficients of the orthogonal loss. Adjust between 0.01 and 1.0.

[0078] (3-2) Embedded Representation Reconstruction The purpose of embedding representation reconstruction is to ensure that each modality shares [the desired representation]. and unique decomposition No original information is lost. For modal... Its shared embedded representation and unique embedding representation splicing is represented as fusion. Build a unique reconstruction decoder for each modality m. The reconstruction decoder typically uses a three-layer MLP structure. The concatenated embedding representation is used to uniquely reconstruct the input modality of the decoder, thus reconstructing the modal input features. , , , in Representing modes The reconstructed features, where N is the number of nodes in the sample, and is related to the original input. Same dimension.

[0079] In this process, for each mode m, the reconstruction process uses mean squared error (MSE) to define the mode embedding representation of the reconstruction loss. , in, It is L2 range The summation of reconstruction losses over all modalities constitutes the embedding representation reconstruction loss. . Embedded representation of the weight coefficients of the reconstruction loss It is recommended that its value be set between 1.0 and 10.0.

[0080] (3-3) Shared embedding representation alignment The shared embeddings alignment aims to achieve explicit alignment of shared extrinsic and neural modalities for the same sample within a unified shared space. Cross-modal contrastive learning is performed using fNIRS modalities as teacher anchors. First, the average extrinsic modality vector for each sample is calculated. in This indicates that the mean value is calculated over the node dimension.

[0081] Secondly, using the fNIRS modality as the teacher anchor for this sample, it is only required that the shared representation of the sample's external modality can be as close as possible to the fNIRS modality of this sample. and with other samples Extend the distance. In a batch (in this invention example set to...) B Within a range of 16, positive sample pairs are formed by the extrinsic mode and its fNIRS mode of the same sample, and negative samples are formed by the fNIRS of the extrinsic mode and other samples in the same batch. Let the similarity be cosine similarity. Temperature coefficient Then the shared embedding alignment loss is, , in Indicates the first One sample, modality The sample-level shared vector consists of a numerator corresponding to the positive pairing of the extrinsic and internal fNIRS modalities of the sample within the same contextual task, and a denominator comparing the negative samples after normalization of all intra-batch fNIRS modalities. The shared embedding representation loss is incorporated into the joint optimization during the training phase, with weights denoted as... Its value is recommended to be between 0 and 0.1. It is not calculated during the deployment phase. .

[0082] The adjacency relationships of nodes within each mode of the shared encoder and the unique encoder described above are determined by the K-nearest neighbor algorithm, where... Values ​​range from 5 to 15. The parameters of the shared encoder are shared across the four modalities, while the unique encoders operate independently, learning the differential features within each modality. (Figure: Number of convolutional layers) Set as 1 to 2 layers, with input dimensions as follows: The output dimension is consistent with the input dimension. During the training phase, four modalities are trained. During the deployment phase, only the external modes are calculated. The fNIRS modality is subsequently derived from the reconstructed neural agent features to obtain corresponding components, in order to maintain consistency with the fusion path during training.

[0083] (4) Generation of neural agent features.

[0084] The MFE-CCAGNN model unit requires alignment of external modality and internal neural modality data display during the training phase. During deployment, it relies solely on the external modality for discrimination. The multimodal feature fusion module receives only the external modality during deployment, where the fNIRS branch is replaced by neural proxy features generated from the external modality.

[0085] In the cross-modal neural agent feature generation stage, the input sample's external behavioral modality ( v , a , t By fNIRS modality-specific reconstruction decoder, the fusion embedding of fNIRS is used. Reconstruct Approximate representation, i.e., neural proxy features Neural agent features replace the real fNIRS modalities in subsequent fusion and discrimination during the deployment phase.

[0086] The neural agent features are reconstructed as follows: in , and These are the fusion embeddings of each external modality. This is the splicing process. Finally, the fNIRS-specific reconstruction decoder is used. Perform agent restructuring from the outside in. and They are of the same dimension.

[0087] The neural surrogate feature reconstruction loss in the above process is obtained by adding the neural surrogate feature MSE loss and the neural surrogate feature geometric loss. The neural surrogate feature MSE loss is defined as follows: At the same time, it is also necessary to ensure the characteristics of the reconstruction and The relative relationships between nodes ensure similarity, avoiding the destruction of the temporal or segment geometry by fitting only point-by-point values. The geometric loss of neural agent features is defined as follows: in This represents the in-sample (node-dimensional) Gram similarity matrix. It is an L2 norm.

[0088] The neural agent feature reconstruction loss is, . in and These are the weights of the neural surrogate feature MSE loss and the neural surrogate feature geometric loss, respectively, and their values ​​are in proportion to... : ≈1:[0.1-1]. The neural agent feature reconstruction loss needs to be included in the joint optimization loss, and the loss weight is denoted as... It is recommended to set it between 0.5 and 5.0.

[0089] During the deployment phase, it is no longer necessary to collect fNIRS modalities; they can be directly generated from external modalities. As neural agent features, the same fNIRS modality branching and fusion process as during training is used to ensure inference path alignment. This ensures that the system can be deployed without relying on the fNIRS modality, enabling the system to automatically align with data input from external modalities.

[0090] (5) Attention-weighted fusion Both during the training and deployment phases, an attention mechanism is ultimately used to assign weights to the shared and unique embeddings, generating a fused final representation that is then input into the downstream classifier module. First, it's crucial to ensure that the dimensions of the shared and unique features are consistent for each modality. Calculate the gating coefficient of the scalar The gating weights are constructed using a lightweight MLP. , in It is the sigmoid function. Based on this, the attention-weighted fusion formula is constructed as follows: . During the training phase, the weighted fusion features of the four modalities are concatenated to form the final representation. During the deployment phase, the neural agent obtained in step (3-3) is used. As isodimensional inputs to the fNIRS modality, neural modalities are obtained respectively.n Neural agent shared embedding representation and unique embedding representation Then calculate Replace the training phase Finally, the weighted fusion features are obtained. .

[0091] The classifier module uses the fused features from the multimodal feature fusion module. The input is a function that takes depression as input and outputs a depression classification result. The classifier module consists of two fully connected layers and a softmax function. The final output is a four-category classification result: healthy, mild, moderate, and severe.

[0092] Its working process is as follows: First, calculate the output of the two fully connected layers in the forward computation. , The output of the two fully connected layers is used The function calculates the probability of four possible levels of depression and finally outputs the index of the depression category. in , These are the weights formed by the fully connected layers 1 and 2, respectively. and That is the corresponding bias. It is a non-linear activation function. In the final classification stage, let the one-hot vector of the true label be y∈{0,1} (the correct class is 1, and the rest are 0), and the class loss weights... , The loss function of the classifier module is designed as follows: . The MFE-CCAGNN model unit needs to be trained before deployment. The specific settings for the training process are as follows: (1) The MFE-CCAGNN model unit needs to perform multi-task joint optimization, and its loss function is set as follows: , in It is the loss of the classifier module. and These are the orthogonal constraint loss and its weight. and These are the neural agent feature reconstruction loss and its weights. and These are the embedding representation reconstruction loss and its weights. and These are the shared embedding alignment loss and its weights. During the deployment phase, it does not participate in the backpropagation or parameter updates of any loss term.

[0093] (2) Optimizer and Learning Rate Strategy: During the training phase, the Adam optimizer is used for parameter updates, with an initial learning rate set to 0.001 and a weight decay term of... To suppress overfitting, the StepLR learning rate scheduler is used, which multiplies the current learning rate by 0.1 every 10 training epochs to gradually reduce the learning rate and promote model convergence. To reduce the overfitting problem caused by small datasets, 5-fold cross-validation is used for training.

[0094] (3) Training cycle and repeatability control: During the training phase, the total number of training rounds of the model was set to 45 rounds, and the training and validation losses were recorded in each round of training. In order to ensure the repeatability of the experimental results, all experiments used a fixed random seed (seed=42) to control the initialization and data partitioning process.

[0095] (4) Evaluation Strategy: During the training phase, in each fold validation, the macro-average F1-score on the validation set is used as the main evaluation metric to select the optimal model parameters; simultaneously, metrics such as Accuracy, Precision, and Recall are recorded for comprehensive performance evaluation. The calculation methods are as follows: , , , , True positive (TP) and true negative (TN) values ​​are defined as the number of correctly classified patients with depression and individuals with HC, respectively. The number of healthy individuals classified as depressed and the number of patients with depression classified as healthy represent false positive (FP) and false negative (FN) values, respectively.

[0096] (5) Training environment configuration: The training process is carried out on a single GPU (such as NVIDIA RTX 3090), based on the PyTorch 1.12 deep learning framework, CUDA version 11.3, and the operating system is Ubuntu 20.04.

[0097] System Deployment: This invention's system is deployed on an integrated assessment terminal or portable psychological screening device with audio and video acquisition, fNIRS signal access, and graphical interaction capabilities. It is applicable to various scenarios such as hospital outpatient clinics, community health service centers, and school counseling rooms. On the deployed device, depression identification results can be displayed and output in the following ways: (1) Local terminal display module: The individual's depression level judgment result and corresponding probability information are displayed in real time on the touch screen at the front of the device in a visual graphic manner, which facilitates rapid on-site assessment and feedback; (2) External interface communication module: Through Bluetooth or WiFi module, the recognition results and basic information are pushed to the hospital information system or remote psychological intervention terminal in a structured JSON format to realize data sharing and remote auxiliary decision-making; (3) Local data recording module: Write the depression level results and their probability distribution for each identification into the local SQLite database for individual historical status management, psychological intervention effect evaluation and subsequent long-term follow-up analysis.

[0098] verify: This embodiment demonstrates the operation of the multimodal depression recognition system based on the MFE-CCAGNN model of the present invention in a real subject task context, including data acquisition, stimulus design, model training process and experimental results.

[0099] 1. Data Acquisition and Preprocessing 1-1 Data Acquisition Environment and Equipment Configuration. Subjects completed the experimental tasks in a quiet, well-lit, and undisturbed environment. In this embodiment, the following equipment was used to achieve simultaneous acquisition of multimodal data: (1) Video acquisition module: equipped with a high-definition camera with a frame rate of 30fps, used to acquire facial video data; (2) Audio acquisition module: using a high-fidelity microphone with a sampling rate of 44.1kHz; (3) fNIRS data acquisition module: The BS-7000 brain functional imaging system is used to cover the prefrontal cortex region, with a sampling frequency of 20Hz and light source wavelengths of 690nm and 830nm.

[0100] Ensure that audio, video, and fNIRS signals are strictly aligned and assigned a unified timestamp.

[0101] It also includes text transcription tools: transcribe audio data into text data.

[0102] 1-2 Data Acquisition Process Steps.

[0103] (1) Subjects wear the fNIRS device and complete position calibration to ensure signal stability; (2) Conduct a resting state data acquisition (1 minute) before the task and record the neural background signals; (3) Entering the task stage, semantic fluency prompts or emotional pictures are presented, and the subjects express themselves verbally while the trimodal data is recorded. (4) Record of resting state after task (1 minute).

[0104] The multimodal data collected synchronously during the above process will be preprocessed and then input into the video feature extraction module, audio feature extraction module, text feature extraction module and fNIRS data feature extraction module of the present invention system.

[0105] Data was collected from 135 participants, and all data were obtained with informed consent and ethical approval.

[0106] 2. Experimental Tasks and Stimulus Situation Design To fully elicit depression-related emotional characteristics, the stimulus materials are divided into the following four scenarios: (1) Positive: Open-ended questions that evoke positive emotions; (2) Neutral: Daily description tasks, used as a control group; (3) Negative: semantic stimuli that induce sadness and depression; (4) Resting state: No task, used for comparison with individual baseline control.

[0107] 3. Classification effect in four stimulus scenarios In this embodiment, the classification performance of the multimodal depression recognition system based on the MFE-CCAGNN model proposed in this invention was quantitatively verified. The classification results under four scenarios (positive, neutral, negative, and resting state) are as follows: Figure 8 The confusion matrix shown is illustrated below. Each row in the confusion matrix represents the true label category, and each column represents the model's prediction result. There are four categories, each corresponding to a different degree of depression.

[0108] In positive scenarios, the system performed well across all categories, with particularly high accuracy rates of 90.91% (10 / 11) for Category 2 (mild) and 91.67% (11 / 12) for Category 4 (severe) samples, demonstrating good stability and anti-interference capabilities.

[0109] In neutral scenarios, apart from a few samples showing prediction bias between neighboring categories, the overall classification accuracy remained at a high level. The severe category (Category 4) was still accurately classified in all cases (11 / 12), indicating that the model has a good ability to identify severely depressed individuals in situations where emotional expression is weak.

[0110] The system performed particularly well under negative situations. Classification errors for all categories were further reduced. Specifically, the accuracy rates for categories 3 and 4 improved to 90.91% and 91.67%, respectively, while the accuracy rate for category 1 (non-depression) reached 87.5% (7 / 8), reflecting that the model is more discriminative in identifying negative emotional stimuli.

[0111] In the resting state, the system maintains stable classification performance, with each category performing in a manner consistent with the neutral state, especially achieving an accuracy of 91.67% on the severe depression sample.

[0112] 4. Performance Analysis of Four Feature Extraction Modules In the deployed state, for each stimulus scenario, the system performance was compared by inputting three external modalities (video, audio, and text), inputting the fNIRS modality individually, and inputting all three external modalities. Figure 9 As shown, the horizontal axis represents the input modality, and the vertical axis represents the performance score. Under different stimulus scenarios (positive, neutral, negative, and resting state), the performance of the single-modality feature extraction model in depression identification exhibits significant differences and limitations. Taking the positive scenario as an example, the accuracy rate of the video modality is 71.43%, which is better than audio (61.9%), text (64.29%), and fNIRS (57.14%). However, in the neutral, negative, and resting state scenarios, the accuracy rates of all single modalities are generally below 70%, showing a simultaneous decline in indicators such as Precision, Recall, and F1-Score. This indicates that a single modality is insufficient to fully capture the full picture of the subject's emotional state, exhibiting problems such as insufficient robustness and poor adaptability to specific scenarios. This further suggests that, in the deployed state, if multiple external modalities are input, it can effectively compensate for the blind spots of single-modality identification in different scenarios, improving the system's ability and stability in recognizing complex emotional states.

[0113] 5. Performance comparison of advanced systems This embodiment also compares the MFE-CCAGNN model unit in the proposed multimodal depression recognition system based on the MFE-CCAGNN model. With the data acquisition unit and data preprocessing unit remaining the same, we replaced the MFE-CCAGNN model unit with a currently mainstream deep learning model to verify its performance effectiveness. Figure 10As shown. These deep learning models mainly include ResNet, CNN1D, LSTM, GRU, CNN-LSTM, and CNN-GRU. Comprehensive comparative results under four stimulus scenarios (positive, neutral, negative, and resting state) show that the MFE-CCAGNN model significantly outperforms other comparative models in key evaluation metrics such as accuracy, precision, recall, and F1-score. In the positive scenario, the MFE-CCAGNN model achieves an accuracy of 85.71% and an F1-score of 0.8511, far exceeding models such as ResNet (64.29%) and GRU (73.81%). In the neutral scenario, the MFE-CCAGNN model achieves an accuracy of 83.33% and an F1-score of 0.8294, an improvement of approximately 14% over CNN-LSTM. In negative situations, the MFE-CCAGNN model performed exceptionally well, achieving an accuracy of 90.48% and an F1-score of 0.9025, nearly 20% higher than ResNet, indicating that the MFE-CCAGNN model has a stronger ability to discriminate under strong negative emotional stimuli. In resting-state situations, the MFE-CCAGNN model also maintained its leading advantage, achieving an accuracy of 83.33% and an F1-score of 0.8294, outperforming all other baseline models.

[0114] 6. Deployment complexity and equipment usability analysis In this embodiment, the multimodal depression recognition system based on the MFE-CCAGNN model is integrated and deployed in a dedicated device that includes a processor unit, a multimodal data acquisition interface, and a model inference engine. The relevant parameters are as follows: (1) The processor unit supports multi-threaded parallel computing (Intel i7 platform). (2) The system inference engine is composed of MFE-CCAGNN model units and is deployed in ONNX format via PyTorch; (3) Modal input interfaces include a camera (for video acquisition), a microphone (for audio acquisition), an fNIRS device (for neural imaging acquisition), and a touch screen (for text input and control). The deployment performance parameters are as follows: (1) Total number of model parameters: 301,601; (2) Inference time (for a single individual data point): 2.01 seconds; (3) Device-side storage overhead: approximately 12.3MB; (4) Memory usage: approximately 184MB, compatible with mainstream mobile or embedded platforms.

[0115] The results demonstrate that the system of this invention, while ensuring multimodal fusion performance, features a compact structure, low computational resource consumption, and is easy to deploy on edge devices, laptops, or mobile medical terminals; it boasts fast inference speed, making it suitable for large-scale rapid identification and real-time feedback of depressive states; and it exhibits good scalability, supporting modality combination configuration and module-level updates. In summary, the depression identification device described in this invention possesses characteristics such as high performance, low complexity, and high adaptability, making it suitable for various application scenarios including grassroots screening, clinical auxiliary diagnosis, and public mental health intervention.

[0116] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. A multimodal depression recognition system based on the MFE-CCAGNN model, characterized in that, It includes a data acquisition unit, a data preprocessing unit, and an MFE-CCAGNN model unit, wherein: The data acquisition unit is used to simultaneously collect video modal data, audio modal data, text modal data, and fNIRS modal data of the subjects in a standardized interview task scenario during the training phase, and to collect only external behavioral modal data in the same task scenario during the deployment phase. The external behavioral modal includes video, audio, and text. The data preprocessing unit includes a video preprocessing unit, an audio preprocessing unit, a text preprocessing unit, and an fNIRS preprocessing unit; The MFE-CCAGNN model unit includes four feature extraction modules: video feature extraction module, audio feature extraction module, text feature extraction module, and fNIRS data feature extraction module, as well as a multimodal feature fusion module and a classifier module; The data acquisition unit provides time-aligned raw data support for the data preprocessing unit; the inputs of each feature extraction module of the MFE-CCAGNN model unit are connected to the outputs of the corresponding preprocessing modules of the data preprocessing unit to extract features from various types of data output by the data preprocessing unit; the multimodal feature fusion module of the MFE-CCAGNN model unit completes explicit alignment, sharing, and unique embedding decoupling and neural proxy feature generation during the training phase, and generates neural proxy features based solely on external behavior during the deployment phase without using neural modalities, and fuses them with external behavior features, and finally the classifier module outputs the depression recognition result.

2. The multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The video preprocessing unit of the data preprocessing unit is used to transform the raw video modal data into a structured, modelable low-level structured descriptor, and finally output video pre-processing data (VPRE). The video preprocessing unit includes a face detection module, a video frame extraction module, and a low-level descriptor extraction module, which are connected in sequence: (1) The face detection module accurately locates the subject's facial area from the input raw video data, removes background and non-facial interference information, and ensures that subsequent processing focuses on the effective area; (2) Video frame extraction module, used to extract frames that can remove redundant frames and unify the face orientation, reduce noise interference caused by pose differences, and perform frame-level sampling; (3) Low-level descriptor extraction module, built using the open-source tool OpenFace, extracts facial key point location, head posture and eye movement data of the subjects during the interview task. These data serve as the biometric features of facial expressions in depression identification. The data output by the low-level descriptor module includes facial expression action unit data, including: Facial motion unit (AUs) intensity: intensity values ​​of 17 motion units; Facial key points: 68 two-dimensional coordinate points, totaling 136 dimensions; Head pose: X / Y / Z 3D position and Pitch / Yaw / Roll rotation angles, totaling 6 dimensions; Eye movement direction: a three-dimensional vector of the left and right eye gaze directions, totaling 6 dimensions; The final output VPRE of the video preprocessing unit is a total of 165 dimensions; The video feature extraction module of the MFE-CCAGNN model unit takes as input the video coding features (VPRE) output by the video preprocessing unit, and its output is... The frame sequence of VPRE is denoted as Each frame The video feature extraction module constructs an emotion activity gating-Transformer structure, which includes an emotion score calculation block, an emotion activity gating weighting block, and a Transformer encoder. The Transformer encoder structure is stacked with two layers, each layer containing four multi-head attention heads, each attention head with a dimension of 32, a total input dimension of 165, and a feed-forward network (FNN) with a dimension of 512. All layers employ residual connections and LayerNorm normalization mechanisms. Its working process is as follows: (1) Calculate the input features and emotion scores using the emotion score calculation block; Each time frame in the VPRE data To represent the structured video behavioral features of this frame, the emotion score for each frame is first calculated. And compressed to the (0,1) interval, in Represents the Sigmoid function; (2) Perform emotional activity gating weighting operation through emotional activity gating weighting block; A gating structure is used to fuse frame features guided by sentiment weights with the original features. ， in , , It's an emotion rating. These are the frame feature representations before and after weighting, respectively, and ReLU(·) is a non-linear activation function; (3) Construct the Transformer encoder; (3.1) First, sequence construction and linear mapping are performed: all frames are constructed into a new video sequence. And use linear mapping to perform embedding dimension transformation to adapt to the Transformer structure, while keeping the input dimension at 165; (3.2) Secondly, multi-head attention modeling of inter-frame dynamic association is performed: a stacked multi-head self-attention mechanism is used to perform context modeling of the emotion-weighted frame sequence. , in It is a stacked calculation. , It is the output mapping matrix, and ， Q, K, and V above represent the query, key, and value vector sequences, respectively. It is a multi-class classification function; (3.3) Finally, nonlinear transformation and residual connection are performed: the output sequence is transformed through a feedforward network and residual connection to obtain the output of the video feature extraction module. , in Indicates a feedforward network. It is layer normalization. This represents the output of the video feature extraction module.

3. The multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The audio preprocessing unit of the data preprocessing unit processes the raw audio speech data and finally outputs audio preprocessing data (APRE). The audio preprocessing unit includes a filtering and noise reduction module, an amplitude normalization module, and a silence data removal module, which are connected in sequence as follows: (1) Filtering and noise reduction module: This module uses a Butterworth bandpass filter to perform frequency domain filtering on the original audio data, filtering out environmental interference signals that are too high or too low, retaining the human voice frequency band in the range of [80Hz, 8000Hz], and filtering out low-frequency noise, current interference and equipment noise; (2) Amplitude normalization module. The function of this module is to perform amplitude normalization processing on the denoised audio data in order to eliminate the influence of volume differences between different individuals or recording environments. (3) Silence data removal module: This module removes silence segments from the normalized audio data to eliminate invalid pauses and background noise interference, retaining valid speech segments. Specifically, the silence data removal module of the audio preprocessing unit removes silence segments using the frame energy determination method, as shown below: , in, For frame energy, It is the input voice data. Represents a speech segment, Represents the total number of speech segments; if the energy of a certain frame satisfies ,in If the threshold for silence is reached, the frame is identified as an invalid silent frame and discarded. (4) Mel power spectrum calculation module. This module is used to calculate the Mel spectrum of the audio data after removing silence. Specifically, this module uses short-time Fourier transform to calculate the power spectrum and uses the Mel filter bank to map to the sensing frequency space to generate a 128-dimensional Mel frequency energy spectrum. The calculation formula is as follows. ， in It refers to the audio frequency of the voice data. Depend on It was converted from; It corresponds to the Mel frequency, i.e., the APRE output by the audio preprocessing unit; The audio feature extraction module of the MFE-CCAGNN model unit takes the APRE output from the audio preprocessing unit as input and outputs audio encoded features as output. ; In the audio feature extraction module, the multi-scale convolutional structure contains three one-dimensional convolutional kernels with sizes of 1×1, 1×3, and 1×5, respectively. Each kernel has 64 channels, and the total number of output channels after concatenation is 192. The convolutional layer is followed by BatchNormalization and ReLU activation functions. To ensure alignment with the original spectrogram in the time dimension, the convolution uses a stride of 1 and symmetrical padding to maintain the same time length. Its working process is as follows: (1) First, for the input APRE data, 1×1, 1×3, and 1×5 convolutional kernels are constructed to extract local details, short-term rhythm, and sliding frequency band changes, respectively. The outputs of the three are concatenated to form a feature fusion representation. , in The input APRE represents the audio data. This represents a one-dimensional convolution along the time dimension, with each branch having 64 output channels. Concatenate according to channel dimension, k takes values ​​of 1, 3, and 5; (2) Channel normalization and activation; Fusion features Perform BatchNorm and ReLU operations to improve the network's nonlinear modeling ability and numerical stability; . in, The activation feature retains the same duration and has 192 channels. (3) Channel residual fusion; Activated features With the original input After addition, the final output of the audio feature extraction module is: .

4. The multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The text preprocessing unit of the data preprocessing unit preprocesses the acquired raw text data and finally outputs text preprocessing data (TPRE); the text preprocessing unit performs data cleaning and standardization. The text feature extraction module of the MFE-CCAGNN model unit takes the TPRE output by the text preprocessing unit as input and outputs the text encoded features as output. ; The text data feature extraction module constructs an N-Transformer structure, which includes multiple Transformer blocks and uses a stacked multi-head attention structure to model the contextual relationships between words. Emotional region attention is introduced in the embedding layer. In the text feature extraction module, the maximum length of the input text sequence is 128. First, the word vectors are projected to a dimension of [dimensional value missing] through the embedding layer. d In a space of 256, a text embedding matrix is ​​formed; the subsequent input is fed into an 8-layer Transformer structure, each Transformer layer contains 8 attention heads, each attention head has a dimension of 32, and the intermediate dimension of the feedforward network is 512. Its working process: (1) Embedding layer and sequence representation; The sequence of TPRE outputs from the text preprocessing unit is represented as follows: ， ， in Represented as the first i The maximum sequence length of the words is d =128; the word vectors are mapped to a dimension of through the embedding layer. From the embedding space, we obtain the embedding matrix. , represented as: in It is the embedding layer, which integrates word or sub-word vectorization with positional encoding; (2) Multi-head self-attention and emotional region attention mask; Embedding matrix The input is fed into a multi-layered stacked Transformer module, each layer containing a multi-head self-attention module and a feedforward network module. The standard multi-head self-attention calculation is as follows: . Q, K, and V are obtained by linear mapping from the input embedding matrix. The dimension is single-head key vector; a region-of-interest attention mechanism based on grammatical error tags is constructed using a mask. in This represents the original attention distribution. This represents the attention area mask constructed from grammatical error tags; 1 indicates retention, and 0 indicates masking. The output for each attention is: Finally, we obtain the context feature representation matrix after Transformer encoding. (3) [CLS] Convergence and Dual-Channel Output; From the context feature matrix Extract the [CLS] flag bit vector Using dual-channel output, including channel-based computation of contextual features, a sentiment semantic representation is obtained. ; Language consistency score of context features Finally, the text features are merged. ， in This represents the output of the text feature extraction module. This represents the sigmoid function.

5. A multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The fNIRS preprocessing unit of the data preprocessing unit processes the fNIRS signals collected from the subjects during the interview task or cognitive stimulation process, and finally outputs fNIRS pre-processing data (NPRE). Specifically, the fNIRS preprocessing unit includes an HbO concentration estimation module, a bandpass filtering and noise reduction module, and an HbO signal baseline correction module, which are connected in sequence as follows: (1) HbO concentration estimation module. This module first performs a logarithmic transformation on the fNIRS light intensity data to obtain the change in optical density, and then uses the modified Bell-Lambert law to estimate the change in hemoglobin concentration. ; (2) Bandpass filtering and noise reduction module. The function of this module is to reduce the noise of the original signal. Bandwidth restriction is applied to filter out non-cognitive components; (3) HbO signal baseline correction module, which performs baseline correction on each channel. Baseline correction is performed by shifting the signal to remove baseline deviation during the resting period. The specific calculation method is as follows. ， in, This represents the task increment of hemoglobin concentration in a certain channel at time t before baseline correction. ), Represented as baseline corrected , Indicates the duration of the resting period, used to estimate the baseline mean; It is an integral variable; The fNIRS feature extraction module of the MFE-CCAGNN model unit takes the NPRE output by the fNIRS preprocessing unit as input and outputs the fNIRS encoded features. ; The fNIRS feature extraction module first calculates the Pearson correlation matrix between different channels to describe the neural collaboration pattern under the task state. Then, to address the problems of high-dimensional redundancy, low-frequency drift and complex connection patterns caused by multi-channel acquisition, it adopts two-dimensional convolution and multi-path pooling connection. The multi-path pooling includes max pooling, average pooling and attention pooling. This module is used for mapping learning and explicit alignment during the training phase, and does not call fNIRS during the deployment phase. The fNIRS feature extraction module of the MFE-CCAGNN model unit includes a Pearson correlation matrix calculation module, a two-dimensional convolutional block, and a multi-path pooling block. Specifically, the input is a 53×53 correlation matrix, the convolutional kernel size is 3×3, the number of channels is 64, and the multi-path pooling block uses three pooling paths, namely max pooling, average pooling, and attention pooling. Its workflow is as follows: (1) The Pearson correlation coefficient of HbO signals between all channels was calculated using the Pearson correlation matrix calculation module; a 53×53 connectivity matrix was output, which represents the co-activation pattern of different brain regions during the subject's task execution, characterizing the individual's neuromodulation features in a depressive state. ， in and This is a channel i and j HbO signal; (2) Use convolution kernels to extract local neural connectivity patterns. (3) The convolutional features are subjected to three parallel pooling methods: max pooling captures high-intensity co-activation regions, average pooling characterizes the overall connectivity stability, and attention pooling enhances the model's ability to focus on key connectivity pathways. Among these, max pooling features... F fnirs-max Average pooling characteristics F fnirs-mean Attention pooling features F fnirs-attn The calculations are as follows: , , ， in It is the importance weight of the attention pooling learning region. The three types of pooling features are concatenated along the channel dimension to form a multi-view neural connectivity graph representation. 。 6. A multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The multimodal feature fusion module of the MFE-CCAGNN model unit takes the outputs of the four feature extraction modules as its input. , , , During the training phase, deep embedding mining and explicit alignment are performed on the four types of features, and optimization is carried out using joint orthogonal constraints of cross-modal contrastive and latent space reconstruction; during the deployment phase, only... , , As input, generate neural agent features corresponding to NFS and fuse them with external behavioral features, outputting fused features. ; The construction of the multimodal feature fusion module is based on the mechanism of shared-unique embedding representation decoupling fusion. The shared-unique embedding representation decoupling fusion mechanism refers to applying orthogonal constraints to shared and unique embeddings within the same modality, and combining embedding representation reconstruction and shared space alignment with fNIRS as teacher anchor points, as well as generating neural agent feature reconstruction from the inside out, thereby achieving a method of robust cross-modal fusion. Its workflow is as follows: (1) Unified representation of dimensions; To perform display alignment and subsequent fusion within the shared space, the four modal features VFS, AFS, TFS, and NFS are first uniformly mapped to the same dimension. This process uses a two-layer MLP to perform dimension alignment on each modality, resulting in an equal-dimensional representation: ， ， in It is an element-wise nonlinear activation function. and These are the weight matrix and bias vector of the first-layer MLP, respectively. and These are the weight matrix and bias vector of the second-layer MLP; the final output is the modality. Node feature matrix Correspondingly, , , , These are the equal-dimensional representations of video, audio, text, and fNIRS modalities, respectively. (2) Embedded representation extraction; (2.1) Extraction of shared embedding representations; Will In a GCN shared encoder with shared input parameters, the output is a shared embedding representation of four modalities. , , and During the training phase, all four modalities are input, and during the deployment phase, the external behavioral modality is used. participate; Shared graph construction: for each modality The image is recorded as follows The set of nodes is , Representing modes m The Input feature representation of each node; edge set The K-Nearest Neighbors (KNN) algorithm is used to measure the Euclidean distance of each node in the semantic space, and each node is connected to its K nearest neighbors in the feature space by an edge. ， in This represents the distance between two objects; the smaller the value, the closer they are. (In the adjacency matrix...) Add self-connects and normalize the adjacency matrix. ， ， in for The degree matrix, It is the identity matrix; for each mode m Using the same set of shared parameters By performing a graph convolution mapping function, the shared embedding representation is obtained as follows: ， in It is an element-wise nonlinear activation function; (2.2) Unique Embedding Representation Extraction Will A unique GCN encoder with non-shared input parameters and a unique embedded representation of the output modality. , , and During the training phase, all four modalities are input, and during the deployment phase, the external behavioral modality is used. participate; Unique graph construction: for each modality The unique picture is recorded. ,in, A unique graph network structure for each mode m, Representing modes The set of nodes below, Indicates the edge connection relationship between nodes; For unique images Unique adjacency matrix Using an independent KNN strategy, the nearest neighbor structure is calculated separately for each modality. The adjacency matrix and its normalized form are as follows: ， ， in Let the node degree matrix be... It is the identity matrix; for each mode The parameters of the unique encoder are not shared; they are used to capture video, semantic, structural, and physiological features specific to each modality. ； in The GCN weight matrix is ​​learned separately for each modality in a unique encoder. It is a non-linear activation function; (3) Shared-unique embedded representation decoupling and fusion mechanism The shared-unique embedding representation decoupling and fusion mechanism aims to construct a joint optimization loss objective around three objectives: orthogonal constraints, embedding representation reconstruction, and shared embedding representation alignment. This objective includes: (3.1) Orthogonal constraints of shared and unique embedding representations Shared Embedded Representation Representing cross-modal general information, modality-unique embedding representation Capture modality-specific differences; impose approximately orthogonal constraints on shared and unique representations of the same modality, making them as linearly independent as possible. ； (3.2) Embedded Representation Reconstruction Embedded representation reconstruction ensures sharing across modalities and unique decomposition Without losing original information; first, for modal Its shared embedded representation and unique embedding representation splicing is represented as fusion. Secondly, a unique reconstruction decoder is constructed for each modality m. The concatenated embedded representation of the input modality is used to uniquely reconstruct the decoder and remodel the modal input features. , ， ， in Representing modes The reconstructed features, where N is the number of nodes in the sample, and is related to the original input. Same dimension; final calculation and The MSE loss is minimized. (3.3) Shared embedding representation alignment Shared embeddings represent alignment that requires explicit alignment of shared extrinsic and neural modalities for the same sample within a unified shared space. Cross-modal contrastive learning is performed using the fNIRS modality as the teacher anchor. First, the average value of the extrinsic modality vector for each sample is calculated. in This indicates that the mean is calculated over the node dimension; secondly, the fNIRS mode is used as the teacher anchor for this sample, requiring only that the shared representation of the sample's external modes be as close as possible to the fNIRS mode of this sample. and with other samples To minimize the impact of fNIRS on the external modality of a sample within a batch, positive sample pairs are formed by combining the external modality with the fNIRS of other samples in the same batch, and negative samples are formed by combining the external modality with the fNIRS of other samples in the same batch. A shared embedding alignment loss is constructed to minimize this loss. The adjacency relationships of nodes within each mode of the shared encoder and the unique encoder described above are determined by the K-nearest neighbor algorithm, where... Values ​​range from 5 to 15; the parameters of the shared encoder are shared across the four modalities, while the unique encoders operate independently, learning the differential features within each modality; the number of convolutional layers is shown in the figure. Set as 1 to 2 layers, with input dimensions as follows: The output dimension is consistent with the input dimension; (4) Generation of neural agent features; The MFE-CCAGNN model unit needs to complete the alignment of external modality and internal neural modality data display during the training phase; during the deployment phase, it only relies on the external modality to complete the discrimination; the multimodal feature fusion module only receives the external modality during the deployment phase, where the fNIRS branch is replaced by neural proxy features generated by the external modality; In the cross-modal neural agent feature generation stage, the input sample's external behavioral modality ( v , a , t By fNIRS modality-specific reconstruction decoder, the fusion embedding of fNIRS is used. Reconstruct Approximate representation, i.e., neural proxy features Neural agent features replace real fNIRS modalities in subsequent fusion and discrimination during the deployment phase. The neural agent features are reconstructed as follows: in , and These are the fusion embeddings of each external modality. It's the splicing process; finally, the fNIRS-specific reconstruction decoder is used. Perform agent restructuring from the outside in. and They are of the same dimension; During the deployment phase, it is no longer necessary to collect fNIRS modalities; they can be directly generated from external modalities. As neural agent features, the same fNIRS modality branching and fusion process as during training is used to ensure inference path alignment; this ensures that the system does not depend on the fNIRS modality during deployment, enabling the system to automatically align data input from external modalities. (5) Attention-weighted fusion Weights are assigned to shared and unique embedding representations for each modality. Calculate the gating coefficient of the scalar The gating weights are constructed using a lightweight MLP. ， in It is the sigmoid function; the attention-weighted fusion formula is, ， During the training phase, the weighted fusion features of the four modalities are concatenated to form the final representation. During the deployment phase, the neural agent obtained in step (3.3) is used. As isodimensional inputs to the fNIRS modality, neural modalities are obtained respectively. n Neural agent shared embedding representation and unique embedding representation Then calculate Replace the training phase Finally, the weighted fusion features are obtained. .

7. A multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The classifier module of the MFE-CCAGNN model unit uses the fused features from the multimodal feature fusion module. The input is depression classification results; the classifier module is composed of two fully connected layers and a softmax function, and finally outputs four classification results, namely healthy, mild, moderate and severe. Its working process is as follows: First, calculate the output of the two fully connected layers in the forward computation. , in , These are the weights formed by the fully connected layers 1 and 2, respectively. and That is the corresponding bias; It is a non-linear activation function; The output of the two fully connected layers is used The function calculates the probability of four possible levels of depression and finally outputs the index of the depression category. In the final classification stage, let the one-hot vector of the true label be y∈{0,1}, with the correct class being 1 and the rest being 0, and the class loss weights... , The classification loss is designed as follows: 。 8. A multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The MFE-CCAGNN model unit needs to be trained before deployment. The training aims to minimize the joint loss function, which is: ， in It is the loss of the classifier module. and These are the orthogonal constraint loss and its weight. and These are the neural agent feature reconstruction loss and its weights. and These are the embedding representation reconstruction loss and its weights. and These are the shared embedding alignment loss and its weights; Specifically, orthogonal constraint loss for, ， in , It is the Frobenius norm. Represents the sum of squares of all elements; weighting coefficients Adjust between 0.01 and 1.0; Specifically, embedding represents the reconstruction loss. for, ， For each mode m, the reconstruction process uses mean squared error (MSE) to define the mode reconstruction loss. ， in, It is the L2 norm; weighting coefficient Its value is set between 1.0 and 10.0; Specifically, shared embedding alignment loss Using contrast loss, ， in Indicates the first One sample, modality Sample-level shared vectors, weight coefficients Set between 0 and 0.1; Specifically, neural agent feature reconstruction loss for, ， in and These are the weights of the neural surrogate feature MSE loss and the neural surrogate feature geometric loss, respectively, and their values ​​are in proportion to... : ≈1:[0.1-1]; Neural agent feature MSE loss for, ， The geometric loss weights for neural agent features are defined as follows: ； Specifically, in the final classification stage, let the one-hot vector of the true label be y∈{0,1}, with the correct class being 1 and the rest being 0, and the class loss weights... The loss function of the classifier module is designed as follows: 。

Images