A multi-modal depression recognition system based on MFE-CCAGNN model

Abstract

The application belongs to the field of artificial intelligence, and provides a multi-modal depression recognition system based on an MFE-CCAGNN model, which comprises a data acquisition unit, a data preprocessing unit and an MFE-CCAGNN model unit. The data acquisition unit synchronously acquires multi-modal data such as video, audio, text and fNIRS of a subject when the subject performs a same interview task. The data preprocessing unit comprises a video preprocessing unit, an audio preprocessing unit, a text preprocessing unit and an fNIRS preprocessing unit. The MFE-CCAGNN model unit comprises a video, audio, text and fNIRS neural signal feature extraction module, a multi-modal feature fusion module and a classification module, and realizes depression recognition and classification result output. The system of the application supports four-level depression degree discrimination, has high recognition accuracy, is light in deployment, has strong interpretability and is suitable for mental health screening and clinical auxiliary evaluation scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, and particularly relates to a multi-modal depression recognition system based on an MFE-CCAGNN model. BACKGROUND

[0002] Depression is a common mental disorder, and its clinical manifestations include symptoms such as low mood and lack of interest. In severe cases, it may even lead to self-harm or suicide. Current clinical diagnosis methods mainly rely on professional interviews by doctors and depression scale assessments, such as the Hamilton Depression Scale (HAMD), the Beck Depression Inventory (BDI), the Self-Rating Depression Scale (SDS), and the Patient Health Questionnaire (PHQ-9). However, these methods are highly dependent on the professional level of doctors and the cooperation of patients, and have strong subjectivity and other shortcomings, which limit the diagnosis efficiency and consistency. At the same time, due to individual cognitive bias, emotional fluctuations, or social expectations, some patients may not accurately reflect their true condition. Even some patients may underestimate or exaggerate their symptoms due to shame or other psychological reasons, leading to misdiagnosis or missed diagnosis during clinical assessment. Therefore, under the condition of limited medical resources, an automatic tool is urgently needed to assist in the clinical screening and recognition of depression.

[0003] With the development of machine learning technology, some methods have attempted to detect depression based on single modalities such as facial expressions and speech. For example, convolutional neural networks are used to analyze facial expressions, or support vector machines and decision trees are used to mine features such as pitch and volume in speech. However, these single-modal methods are often susceptible to data occlusion, environmental noise, and situational differences, resulting in unstable detection performance, insufficient generalization ability and personalized adaptability, making it difficult to achieve satisfactory results in actual complex scenarios.

[0004] To improve accuracy and robustness, some studies have introduced external behavior (such as video, audio, and text) multi-modal data for multi-modal fusion, which is relatively easy to obtain. In terms of methodology, convolutional networks, Transformers, or gated recurrent structures are used for joint modeling, which has certain improvement in recognition performance compared to single modalities.

[0005] At the same time, some studies use electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS) to detect depression, which can reflect the state of neural response, and the performance is relatively outstanding.

[0006] However, the method of using external behavior multi-modal fusion ignores the correspondence between external behavior and individual internal neural state. For example, the same "smile" may be a social response in a greeting scenario, represent positive emotions in a happy scenario, and imply suppressed emotions in a family conflict dialogue. Without alignment with neural responses, external cues are prone to contextual ambiguity. In addition, relying solely on physiological data of neural responses has obvious limitations. First, data collection is strongly dependent on equipment and environment, such as high cost and sensitivity to motion artifacts for fMRI, and EEG is easily affected by noise and electrode contact, making it difficult to promote in large-scale population screening. Second, physiological indicators are affected by individual differences, current state, and cross-device differences, limiting the consistency and comparability across populations and scenarios. Third, the use of single neural signals lacks alignment with specific external behaviors (and context), making it difficult to determine the true emotional meaning of the same physiological changes, resulting in insufficient explanation and false positives or false negatives.

[0007] In summary, existing technologies either focus only on external behavior and ignore internal neural state, or rely solely on neural physiology and deviate from real interaction context, making it difficult to balance stability, explainability, and deployability. Therefore, there is an urgent need for a unified interview task paradigm that can simultaneously map external behavior modalities and internal neural responses to eliminate the lack of internal neural modal support for external behavior modalities and to resolve the contextual ambiguity of internal neural modalities, thereby improving the reliability of depression recognition. SUMMARY

[0008] The present application aims to overcome the limitations of existing depression recognition systems that rely solely on external behavior or only on neural physiological signals, and proposes a multi-modal depression recognition system based on MFE-CCAGNN (Multimodal Feature Extraction and Cross-Modal Co-Attention with Graph Neural Networks). The system synchronously collects external behavior modalities (video, audio, text) and functional near-infrared spectroscopy (fNIRS) neural responses during the training phase, enabling explicit alignment of external behavior and internal neural responses. In the deployment phase, the system can automatically generate neural proxy features from external behavior without relying on neural modalities, thereby reducing device dependency while maintaining explainability and improving the stability and generalizability of screening.

[0009] In the system of the application, the explicit alignment of external behavior modalities (video, audio, text) and fNIRS neural responses enables a deep understanding of the internal-external mapping relationship, so that in the future in the context of missing neural modalities such as large-scale depression recognition, the system can automatically generate neural agent features corresponding to neural representations based on external behavior only to participate in discrimination, thereby maintaining interpretability while reducing device dependence and improving the stability and promotion of screening. Considering the requirements of interview scenarios for natural interaction and deployment convenience, fNIRS has engineering advantages such as portability, tolerance to slight head movement, easy deployment with camera and audio pickup devices, etc., and is suitable as a teacher anchor source for system training period. The application accordingly proposes a technical solution of a specific depression recognition system.

[0010] The technical solution of the application is as follows:

[0011] A multi-modal depression recognition system based on an MFE-CCAGNN model, comprising a data acquisition unit, a data preprocessing unit, and an MFE-CCAGNN model unit, wherein:

[0012] The data acquisition unit is used to synchronously acquire video modal data, audio modal data, text modal data, and fNIRS modal data of a subject in a standardized interview task scenario during the training phase, and to acquire only external behavior modal data in the same task scenario during the deployment phase, the external behavior modalities including video, audio, and text.

[0013] The data preprocessing unit includes a video preprocessing unit, an audio preprocessing unit, a text preprocessing unit, and an fNIRS preprocessing unit.

[0014] The MFE-CCAGNN model unit includes four feature extraction modules: a video feature extraction module, an audio feature extraction module, a text feature extraction module, and an fNIRS data feature extraction module, as well as a multi-modal feature fusion module and a classifier module.

[0015] The data acquisition unit provides time-aligned raw data support for the subsequent data preprocessing unit; the inputs of each feature extraction module of the MFE-CCAGNN model unit are respectively connected to the outputs of the corresponding preprocessing modules of the data preprocessing unit, and the feature extraction is performed on each type of data output by the data preprocessing unit; the multi-modal feature fusion module of the MFE-CCAGNN model unit completes explicit alignment, shared and unique embedding decoupling, and neural agent feature generation during the training phase, and generates neural agent features based on external behavior only without using neural modalities during the deployment phase, and then fuses the neural agent features with external behavior features, and finally outputs the depression recognition result by the classifier module.

[0016] The design principle of the above system technical solution is as follows:

[0017] By designing a unified data acquisition unit, four types of modal data (video, audio, text, and fNIRS) are synchronously collected under standardized interview tasks, ensuring time consistency and psychological state correspondence. Based on the synchronous acquisition mechanism during the training period, the external behavior and internal neural response are placed in the same time context, overcoming the problems of incomplete expression and insufficient robustness of single-modal and external behavior-dependent multi-modal methods for depression-related information.

[0018] Meanwhile, the supporting video, audio, text, and fNIRS four data preprocessing units respectively complete the structured feature extraction of the video modality, the speech denoising standardization of the audio modality, the language text cleaning of the text modality, and the neural signal correction processing of the fNIRS modality, significantly improving the quality and usability of multi-modal data.

[0019] Finally, the MFE-CCAGNN model unit includes video, audio, text, and fNIRS neural signal feature extraction modules, multi-modal feature fusion modules, and classifier modules. Among them, the video feature extraction module adopts an emotion gating mechanism-Transformer. The audio feature extraction module adopts multi-channel convolution for feature extraction. The text feature extraction module adopts N-Transformer for feature extraction. The fNIRS feature extraction module adopts multi-scale convolution for feature extraction. The multi-modal feature fusion module adopts a shared-unique embedding representation decoupling fusion mechanism based on graph convolutional neural network and attention fusion strategy for feature fusion, realizing joint representation modeling of behavior-language-neural multi-dimensional pathways. Finally, the classifier module realizes depression recognition and classification result output.

[0020] The beneficial effects of the present application are:

[0021] The present application system supports four-level depression degree discrimination, has high recognition accuracy, is easy to deploy, has strong interpretability, and is suitable for psychological health screening and clinical auxiliary evaluation scenes. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The present application system is a whole design schematic diagram;

[0023] Figure 2 The present application system is a whole design schematic diagram;

[0024] Figure 3 The present application system is a whole design schematic diagram;

[0025] Figure 4 The present application system is a whole design schematic diagram;

[0026] Figure 5An implementation schematic diagram of a text feature extraction module of a text preprocessing unit and an MFE-CCAGNN model unit;

[0027] Figure 6 An implementation schematic diagram of an fNIRS feature extraction module of an fNIRS preprocessing unit and an MFE-CCAGNN model unit;

[0028] Figure 7 An implementation schematic diagram of a multi-modal feature fusion module of an MFE-CCAGNN model unit;

[0029] Figure 8 Prediction results of a depression recognition system under four stimulation tasks;

[0030] Figure 9 Comparison of classification performance of different modalities in the system input of a depression recognition system in the deployment stage under four stimulation types;

[0031] Figure 10 Performance comparison between an MFE-CCAGNN model unit and current mainstream deep learning models in the deployment stage of a depression recognition system. DETAILED DESCRIPTION

[0032] The technical solutions provided by the present application will be further described below with reference to specific embodiments and the accompanying drawings. The advantages and features of the present application will become clearer in combination with the following description.

[0033] As shown in Figure 1 :

[0034] A multi-modal depression recognition system based on an MFE-CCAGNN model, comprising a data acquisition unit, a data preprocessing unit, and an MFE-CCAGNN model unit, wherein:

[0035] The data acquisition unit is configured to synchronously acquire video modal data, audio modal data, text modal data, and fNIRS modal data of a subject under a standardized interview task scenario in a training stage, and to acquire only external behavior modal (video, audio, and text) data under the same task scenario in a deployment stage.

[0036] The data preprocessing unit comprises a video preprocessing unit, an audio preprocessing unit, a text preprocessing unit, and an fNIRS preprocessing unit.

[0037] The MFE-CCAGNN model unit mainly comprises four feature extraction modules, i.e., a video feature extraction module, an audio feature extraction module, a text feature extraction module, and an fNIRS data feature extraction module, as well as a multi-modal feature fusion module and a classifier module.

[0038] The data acquisition unit provides time-aligned raw data support for the subsequent data preprocessing unit. The input of each feature extraction module of the MFE-CCAGNN model unit is connected with the output of the corresponding preprocessing module of the data preprocessing unit, respectively, to extract features from various data output by the data preprocessing unit. The multi-modal feature fusion module of the MFE-CCAGNN model unit completes explicit alignment, shared and unique embedding decoupling and neural agent feature generation in the training stage. In the deployment stage, the neural agent feature can be generated based on the external behavior only without using the neural modality, and then fused with the external behavior feature. Finally, the classifier module outputs the depression recognition result.

[0039] As shown in Figure 2

[0040] The related devices involved in the data acquisition unit are all existing devices, and the device hardware itself is not the technical innovation point of the technical solution of the present application.

[0041] Specifically, the video modality data in the data acquisition unit mainly refers to the facial video data of the subject during the interview task. The facial video data can further extract the biological characteristic data of the subject during the interview task, such as facial key point position, head posture, eye movement behavior, etc. (the extraction process will be completed in the video preprocessing unit of the data preprocessing unit). Depressed individuals usually show poor facial expressions, reduced positive expressions, and enhanced negative expressions. Common Action Units (AUs) combinations may appear mode changes (such as a significant decrease in smile frequency). In terms of facial geometry, there is a tendency to droop the corners of the mouth, relax the eyelids, and reduce facial muscle tension, which can be quantified by facial key point trajectories. In terms of head posture, depressed individuals may often exhibit behaviors such as lowering their heads, avoiding eye contact, and reducing head movement amplitude, and related information can be captured by pitch, yaw, and roll angles. There are also typical differences in eye movement behavior, such as shortened fixation time, eye avoidance, and glassy eyes, and the direction of eye movement changes relatively slowly.

[0042] ​Specifically, the audio modality data in the data acquisition unit mainly refers to the sound of the subject answering questions during the interview task. In the process of language communication, the common characteristics of patients with depression are slow speech, flat tone, decreased voice energy, increased hesitation delay, etc. These non-semantic acoustic features can be used to reflect the psychological state without relying on semantic understanding. The dynamic time continuity of audio data can capture the emotional changes of patients in the communication process in real time. For example, the tone of depressed patients is usually more monotonous, lacking of ups and downs, the speech rate is significantly slowed down, the pause interval is prolonged, the sound intensity shows a continuous low-energy voice output, and even the voice is interrupted and the sound is trembling or unstable. The above differences make the audio modality have the ability to represent the state related to depression.

[0043] Specifically, the text modality data in the data acquisition unit mainly refers to the semantic content of the subject answering questions during the interview task. The language content of individuals with depression may show patterns such as an increase in the proportion of negative emotional words, an increase in the use of first-person pronouns, a decrease in active sentences, and a relative lack of emotional expression. Changes in the above semantic and psychological characteristics can be modeled and quantified through the text modality, such as changes in the frequency of use of words such as "disappointment", "suffering", "helplessness", changes in the proportion of first-person references, and sentence structure and fluency indicators, to reflect trends such as cognitive negativity, emotional avoidance, and expression simplification. In this application, the text modality data of the data acquisition unit is used for identification, not for clinical differential diagnosis between different mental disorders.

[0044] Specifically, the fNIRS modality data in the data acquisition unit mainly refers to the infrared light intensity change data of the subject's cerebral cortex region collected by the functional near-infrared spectroscopy device during the interview task, cognitive stimulation, or emotional induction process. In terms of neurophysiological mechanisms, patients with depression generally show reduced neural activation in the prefrontal cortex (such as the dorsolateral prefrontal cortex DLPFC, the medial prefrontal cortex mPFC, and other regions) during tasks such as speech expression, emotion arousal, or memory recall. In terms of neural representation, depression may be accompanied by reduced activation, low synchronization of functional networks, and deactivation of the cortex, showing insufficient response to external task stimuli and decreased neural processing efficiency. Therefore, fNIRS can be used to measure the task state of individuals with depression and obtain their abnormal response patterns in the cerebral cortex in a timely manner. Under task-induced conditions, the hemoglobin (HbO) task increment (ΔHbO) of individuals with depression may be lower than that of healthy controls, and the peak appearance time may be delayed. In this application, fNIRS modality is used as a teacher anchor point in the training stage to explicitly align with external behavior modalities and learn the mapping relationship between external behavior and internal neural state. In the deployment stage, fNIRS is no longer collected, and the neural agent features generated by external behavior are used for discrimination.

[0045] The above-mentioned four modalities of video, audio, text and fNIRS are complementary in information dimension and data source, respectively representing two key feature sources of external behavior performance (video, speech, text) and internal neurophysiological state. The joint use of the four modalities provides support for synchronous alignment and mapping learning in the training phase, and maintains the approximate representation of the "behavior-neuro" relationship through the neuroproxy features generated by external behavior in the deployment phase, so as to balance the explainability and deployability in the identification task.

[0046] The algorithm design of each module is described below.

[0047] As shown in Figure 3 :

[0048] The video preprocessing unit of the data preprocessing unit is used to convert the original video modality data into structured and modelable low-level structured descriptors, and finally output video preprocessing data (Video PRE-processing data, VPRE). The preprocessing procedure is applicable in both the training phase and the deployment phase of the system.

[0049] Specifically, the video preprocessing unit includes a face detection module, a video frame interception module, and a low-level descriptor extraction module, which are sequentially connected.

[0050] (1) The face detection module accurately locates the face region of the subject for the input raw video data, eliminates background and non-face interference information, and ensures that the subsequent processing focuses on the effective area;

[0051] (2) The video frame interception module is used to intercept redundant frames and unify the face orientation, reduce the noise interference caused by posture differences, and perform frame-level sampling at an interval of 0.1 seconds;

[0052] (3) The low-level descriptor extraction module is constructed using the open-source tool OpenFace to extract data such as the face key point position, head posture, and eye movement behavior of the subject during the interview task (these data can be used as biological features of facial expressions in depression recognition). The data output by the low-level descriptor module includes facial expression action unit data, including:

[0053] Facial action unit (AU) intensity: intensity values of 17 action units;

[0054] Facial key points: 68 two-dimensional coordinate points, a total of 136 dimensions;

[0055] Head posture: X / Y / Z three-dimensional position and Pitch / Yaw / Roll rotation angle, a total of 6 dimensions;

[0056] Eye gaze direction: 3D vector of gaze direction of left and right eyes, total 6 dimensions.

[0057] The video pre-processing unit finally outputs VPRE, which is a total of 165 dimensions.

[0058] As shown in Figure 3 :

[0059] The video feature extraction module of the MFE-CCAGNN model unit, whose input is the video coding feature VPRE output by the video pre-processing unit, and its output is . For the convenience of subsequent description, the frame sequence of VPRE is denoted as , where each frame .

[0060] The typical facial behavior changes of patients with depression are often characterized by "non-severe type", "low activity" and "slow evolution", such as decreased facial muscle tension, reduced number of smiles, avoidance of eye contact, and shortened gaze time. The expression frequency of these behavior characteristics is low, the duration is short, and the frame-level distribution is uneven, which leads to the dilution of common models (such as traditional Transformer or Bi-LSTM) in the encoding process by a large number of neutral frames, so that the mood-related changes cannot be accurately aggregated.

[0061] To solve the conflict between "mood frame sparsity" and "inter-frame information interference", we focus on the micro-expression fluctuations and abnormal expression of facial action units closely related to the depression state, and the video feature extraction module constructs an emotion activity gating-Transformer structure, which includes an emotion score calculation block, an emotion activity gating weighting block, and a Transformer encoder. The Transformer encoder structure is stacked with 2 layers, each layer contains 4 Multi-HeadAttention, the dimension of each attention head is 32, the total input dimension is 165, the Feed-Forward Network (FNN) dimension is 512, and both use residual connection and LayerNorm normalization mechanism. This module uses

[0062] Its working process is as follows:

[0063] (1) Calculate the input features and emotion scores by the emotion score calculation block.

[0064] Each time video frame in the VPRE data represents the structured video behavior features of the frame, first calculate the emotion score of each frame and compress it to the interval (0, 1),

[0065]

[0066] wherein represents a Sigmoid function.

[0067] (2) The emotion activity gating weighting operation is performed by an emotion activity gating weighting block.

[0068] The emotion weight guided frame features and the original features are fused using a gating structure,

[0069] ,

[0070] wherein , , is an emotion score, and are frame feature representations before and after weighting respectively, and ReLU(·) is a nonlinear activation function.

[0071] (3) A Transformer encoder is constructed.

[0072] (3.1) First, sequence construction and linear mapping are performed: all frames are constructed into a new video sequence, and linear mapping is used for embedding dimension conversion to adapt to the Transformer structure, and the input dimension is kept as 165.

[0073] (3.2) Second, multi-head attention is used to model inter-frame dynamic association: a stacked multi-head self-attention mechanism is used to model the context of the emotion weighted frame sequence,

[0074] ,

[0075] wherein is a stacked calculation, , is an output mapping matrix, and

[0076] .

[0077] The above Q, K, and V are query (Query), key (Key), and value (Value) vector sequences, is a multi-classification function. In this embodiment, the number of attention heads , the dimension of a single head is , and the dimension after splicing is .

[0078] (3.3) Finally, nonlinear conversion and residual connection are performed: the output sequence is constructed by a feedforward network and residual connection for nonlinear conversion, and the video feature extraction module output is obtained,

[0079] ,

[0080] wherein represents a feedforward network, is layer normalization, represents an output of a video feature extraction module.

[0081] As Figure 4 shown:

[0082] The audio preprocessing unit (hereinafter referred to as the audio preprocessing unit) of the data preprocessing unit processes the original audio speech data and finally outputs audio preprocessing data (Audio PRE-processing data, APRE). The preprocessing procedure is applicable in both the training stage and the deployment stage.

[0083] The audio preprocessing unit includes a filter denoising module, an amplitude normalization module, and a silent data removal module, which are sequentially connected as follows:

[0084] (1) Filter denoising module. This module uses a Butterworth band-pass filter to perform frequency domain filtering on the original audio data, filtering out environmental interference signals with excessively high or low frequencies, retaining the human voice frequency band within [80Hz, 8000Hz], and filtering out low-frequency noise, current interference, and device noise.

[0085] (2) Amplitude normalization module. The function of this module is to perform amplitude normalization on the denoised audio data to eliminate the impact of volume differences in different individuals or recording environments. Specifically, the amplitude normalization module of the audio preprocessing unit can resample the original collected speech to 16kHz and perform amplitude normalization to standardize the amplitude interval to [-1, 1].

[0086] (3) Silent data removal module. The function of this module is to remove the silent segments of the normalized audio data to exclude invalid pauses and background noise interference and retain the valid speech segments. Specifically, the silent data removal module of the audio preprocessing unit removes silent segments through a frame energy determination method, which can be expressed as follows:

[0087] ,

[0088] wherein, is the frame energy, is the input speech data, represents a speech segment, represents the total number of speech segments. If a frame energy satisfies , wherein is the threshold of silence, the frame is identified as an invalid silent frame and is removed.

[0089] (4) Mel power spectrum calculation module. The role of this module is to calculate the Mel-spectrogram of the audio data after removing the silence. Mel-spectrogram can reflect the rhythm, tone, sound intensity and other change information in the speech signal, and reflect the abnormal speech patterns such as slow speech, flat tone and energy decline commonly seen in individuals with depression. Specifically, this module uses short-time Fourier transform to calculate the power spectrum, and uses Mel filter bank to map to the perceptual frequency space to generate a 128-dimensional Mel frequency energy spectrum, and the calculation formula is as follows,

[0090] .

[0091] wherein is the audio frequency of the speech data, converted from ; is the corresponding Mel frequency, i.e. the APRE output by the audio preprocessing unit.

[0092] As shown in Figure 4 :

[0093] The audio feature extraction module (hereinafter referred to as the audio feature extraction module) of the MFE-CCAGNN model unit, the input of which is the APRE output by the audio preprocessing unit, and the output is the audio coding feature .

[0094] Depression patients often have slow speech, flat tone, low sound intensity, weak frequency fluctuation and other nonlinear change characteristics in natural speech communication. The Mel-spectrum obtained by the audio preprocessing unit often shows sliding frequency change, rhythm imbalance and energy density drop and other micro-feature distribution. In order to capture these multi-scale speech features, the audio feature extraction module uses different scale convolution kernels in parallel to extract local details, short-term rhythm and frequency band fluctuation information respectively, and splices to form a fusion spectrum domain representation. At the same time, using the channel residual structure, through the fusion of the original input and the multi-scale convolution output, the spectral graph structure consistency is maintained, and the anti-interference robustness of the deep feature extraction stage is improved.

[0095] In the audio feature extraction module, the multi-scale convolution structure contains three one-dimensional convolution kernels with sizes of 1x1, 1x3 and 1x5 respectively, and the channel number of each convolution kernel is 64. The total number of output channels after splicing is 192. After the convolution layer, BatchNormalization and ReLU activation function are connected. In order to ensure the alignment with the original spectrum graph in the time dimension, the convolution uses stride 1 and symmetric padding (same padding), so as to keep the time length unchanged.

[0096] Its working process is:

[0097] (1) Firstly, the input of the APRE data is processed to construct 1x1, 1x3 and 1x5 convolution kernels to extract local details, short rhythms and sliding band changes, respectively, and the three outputs are spliced to form a feature fusion representation,

[0098] ,

[0099] wherein represents the input APRE of the audio data, represents one-dimensional convolution along the time dimension, and the output channel number of each branch is 64, The channels are spliced, and k is 1, 3 and 5.

[0100] (2) Channel normalization and activation. The fused features are subjected to BatchNorm and ReLU operations to improve the network's nonlinear modeling ability and numerical stability.

[0101] .

[0102] wherein, is the activated feature, the time length remains unchanged, and the channel number is 192.

[0103] (3) Channel residual fusion. The activated feature is added to the original input , and the output of the audio feature extraction module is .

[0104] As shown in Figure 5 :

[0105] The text preprocessing unit (hereinafter referred to as the text preprocessing unit) of the data preprocessing unit preprocesses the obtained original text data and finally outputs text preprocessing data (Text PRE-processing data, TPRE).

[0106] The text preprocessing unit is a data cleaning and standardization processing module, which is a python script file. The file runs in the python3.8 environment, which can clean and standardize the text data. The processing flow of the script includes removing mood words (such as "um" and "ah"), non-verbal noise marks (such as medical staff prompts), meaningless repetition (such as "I I I feel" to "I feel"), and abnormal character or abnormal sentence correction to eliminate redundant interference information and improve the purity and stability of semantic modeling. Language phenomena such as wrong words, sentence breaks and repeated interruptions are retained without automatic rewriting or polishing at the semantic level.

[0107] As shown in Figure 5 :

[0108] The text feature extraction module (hereinafter referred to as the text feature extraction module) of the MFE-CCAGNN model unit takes the TPRE output by the text preprocessing unit as input and outputs the text encoded features as output. .

[0109] In natural communication, patients with depression often exhibit characteristics such as sparse semantic structure, disordered language expression, redundant interjections, low density of emotional words, and high rate of grammatical errors. They are also prone to linguistic noise phenomena such as pauses, interfering words, and structural omissions.

[0110] To address this, the text data feature extraction module constructs an N-Transformer structure, which includes multiple Transformer blocks and employs a stacked multi-head attention structure to model the contextual relationships between words. Emotional region attention is introduced into the embedding layer to enhance the position of negative emotion words and first-person pronouns, strengthening the model's ability to perceive depressive tendencies and output stable features. In the text feature extraction module, the maximum length of the input text sequence is 128. First, the word vectors are projected onto a dimension of [dimension not specified in the original text]. d In a space of 256, a text embedding matrix is ​​formed; the subsequent input is fed into an 8-layer Transformer structure, each Transformer layer contains 8 attention heads, each attention head has a dimension of 32, and the intermediate dimension of the feedforward network is 512.

[0111] Its working process:

[0112] (1) Embedding layer and sequence representation. The sequence representation of the Text Preprocessing Unit (TPRE) output is as follows:

[0113] , ,

[0114] in It can be represented as the first i The maximum sequence length of the words is d =128. The word vectors are mapped to an embedding space of dimension d through an embedding layer, resulting in the embedding matrix. , represented as:

[0115]

[0116] in It is an embedding layer that can achieve the fusion of word or subword vectorization and positional encoding.

[0117] (2) Multi-head self-attention and emotion region attention masking. Embedding matrix Input into the multi-layer stacked Transformer module (8 layers identified by the invention), each layer contains multi-head self-attention module and feedforward network module (intermediate layer dimension is 512), the standard multi-head self-attention calculation is:

[0118] .

[0119] Where Q, K, V are obtained by linear mapping of input embedding matrix, is the single-head key vector dimension. Wherein for global attention, only attention area attention is needed, therefore using mask to construct attention area attention based on syntax label construction,

[0120]

[0121] Where represents the original attention distribution, represents the attention area mask constructed by the syntax label (1 represents retention, 0 represents shielding).

[0122] The output of each attention is , and finally the context feature representation matrix after Transformer encoding is obtained,

[0123]

[0124] (3) [CLS] convergence and double-channel output. From the context feature matrix extract the [CLS] flag vector (representing global semantic summary), using double-channel output including channel calculating emotional semantic representation of context features .

[0125] The language consistency score of the context feature , finally the text features are merged,

[0126] ,

[0127] Where represents the output of the text feature extraction module, represents the sigmoid function.

[0128] As shown in Figure 6 :

[0129] The fNIRS preprocessing unit of the data preprocessing unit (hereinafter referred to as the fNIRS preprocessing unit) processes the fNIRS signals collected from the subject during the interview task or cognitive stimulation process, and finally outputs fNIRS preprocessing data (fNIRS PRE-processing data, NPRE).

[0130] Specifically, the fNIRS preprocessing unit includes an HbO concentration estimation module, a band-pass filtering and noise reduction module, and an HbO signal baseline correction module, which are sequentially connected as follows:

[0131] (1) HbO concentration estimation module. This module first performs logarithmic transformation on the fNIRS light intensity data to obtain the optical density change, and uses the modified Beer-Lambert law to estimate the change in hemoglobin concentration .

[0132] (2) Band-pass filtering and noise reduction module. The function of this module is to limit the frequency band of the original and filter out non-cognitive related components. Specifically, the band-pass filtering and noise reduction module mainly uses an FIR band-pass filter to limit the frequency range to 0.01-0.1 Hz to retain the low-frequency components related to the task induction and suppress low-frequency drift and high-frequency noise.

[0133] (3) HbO signal baseline correction module. This module can perform baseline correction processing on each channel to shift the signal and remove the baseline deviation during the resting period. The specific calculation method is as follows,

[0134] ,

[0135] wherein, represents the hemoglobin concentration task increment of a certain channel at time t before baseline correction , represents the hemoglobin concentration task increment of a certain channel at time t after baseline correction , represents the resting period length, which is used to estimate the baseline mean. is an integral variable.

[0136] As shown in Figure 6 :

[0137] The fNIRS feature extraction module of the MFE-CCAGNN model unit (hereinafter referred to as the fNIRS feature extraction module) has NPRE output by the fNIRS preprocessing unit as input, and outputs fNIRS encoding features .

[0138] The activation level of the prefrontal cortex and cingulate cortex in individuals with depression during emotional processing and task execution may be reduced, and the functional connection between brain regions presents desynchronization, non-homogeneous activation pattern. In fNIRS data, this is reflected in the weakening of the correlation between HbO signals in different channels and the decline in connection stability. Therefore, the fNIRS feature extraction module first calculates the Pearson correlation matrix between different channels to describe the neural coordination pattern in the task state. Subsequently, in view of the problems such as high-dimensional redundancy, low-frequency drift and connection mode complexity caused by multi-channel acquisition, two-dimensional convolution, multi-path pooling (including maximum pooling, average pooling and attention pooling) connection is adopted. The module is used for mapping learning and explicit alignment in the training stage, and fNIRS is not called in the deployment stage.

[0139] The fNIRS feature extraction module of the MFE-CCAGNN model unit, the structure of which comprises a Pearson correlation matrix calculation module, a two-dimensional convolution block and a multi-path pooling block. Specifically, the input is a 53×53 correlation matrix (the fNIRS device output data format is 53 channel data), the convolution kernel size is 3×3, the channel number is 64, and the multi-path pooling block adopts three pooling paths, which are maximum pooling, average pooling and attention pooling.

[0140] The workflow is as follows:

[0141] (1) The Pearson correlation coefficient of the HbO signal between all channels is calculated by using the Pearson correlation matrix calculation module. A 53×53 connection matrix is output, which can represent the coordinated activation pattern between different brain regions of the subject during task execution, and represent the neural regulation characteristics of the individual in the depressive state,

[0142] ,

[0143] Among them, and are the HbO signals of channels i and j .

[0144] (2) Local neural connection patterns are extracted using convolution kernels,

[0145]

[0146] (3) The convolution features are three-parallel-pooled, the maximum pooling captures the high-intensity coordinated activation area, the average pooling represents the overall connection stability, and the attention pooling enhances the attention ability of the model to the key connection path, wherein the maximum pooling feature F fnirs-max , the average pooling feature F fnirs-mean , and the attention pooling feature F fnirs-attnare calculated as follows, respectively:

[0147] , , ,

[0148] wherein is the attention-pooling learning area importance weight, . The three types of pooling features are spliced in the channel dimension to form a multi-view neural connection graph representation,

[0149] .

[0150] As shown in Figure 7 :

[0151] The multi-modal feature fusion module, whose input is the output of the above four feature extraction modules , , , . In the training stage, the four types of features are deeply embedded, mined and explicitly aligned, and the optimization of cross-modal comparison and latent space reconstruction joint orthogonal constraint is performed; in the deployment stage (missing neural modality), only , , is input to generate the neural agent feature corresponding to NFS and fuse with the external behavior feature, and the weighted fusion feature is output.

[0152] The output features of the four feature extraction modules represent the significant features of the modalities, but the expression strength, duration and manifestation of each modality feature often differ. If only splicing or uniform embedding strategy is used, it is easy to cause interference between modalities, suppress key signals or introduce redundant information. On the other hand, multi-modal signals under the same task context contain both common representations shared with other modalities and unique representations only manifested in the current modality. For example, in the question and answer "do you feel happy?", the video modality contains both micro-expression and other unique embedded representations, and also contains common emotional responses consistent with speech and text. Accordingly, the fusion module adopts a shared-unique embedded representation decoupling fusion mechanism to ensure the cooperation of common information while preserving the differentiated clues of each modality.

[0153] To this end, in view of the complex characteristics of the coexistence of inter-modal information commonality and expression difference in the multi-modal data in the depression recognition task, the multi-modal feature fusion module is constructed based on a shared-unique embedding representation decoupling fusion mechanism to realize robust fusion of multi-modal features. The shared-unique embedding representation decoupling fusion mechanism refers to imposing orthogonal constraints on shared and unique embeddings within the same modality, combining embedding representation reconstruction, and fNIRS as a teacher anchor point for shared space alignment, and generating neural proxy feature reconstruction from the inside to the outside, thereby realizing a cross-modal robust fusion method.

[0154] The workflow is as follows:

[0155] (1) Dimension uniform representation.

[0156] To perform display alignment and subsequent fusion in the shared space, first, the four modal features VFS, AFS, TFS, and NFS are uniformly mapped to the same dimension . This process uses two-layer MLP to perform dimension alignment on each modality to obtain equal dimension representation:

[0157] , ,

[0158] wherein is an element-wise nonlinear activation function, and are the weight matrix and bias vector of the first layer MLP, respectively, and are the weight matrix and bias vector of the second layer MLP, respectively. The final output is the node feature matrix of the modality . Correspondingly, , , , are the equal dimension representations of the video, audio, text, and fNIRS modalities, respectively.

[0159] (2) Embedding representation extraction

[0160] (2-1) Shared embedding representation extraction

[0161] Input the parameter shared GCN shared encoder, and output the shared embedding representations , , , and of the four modalities. In the training stage, all four modalities are input, and in the deployment stage, the external behavior modality is used.

[0162] Shared graph construction: for each modality ​The image is recorded as follows The set of nodes is , Representing modes m The The input feature representation of each node. Edge set. The K-Nearest Neighbors (KNN) algorithm is used to measure the Euclidean distance of each node in the semantic space, and each node is connected to its K nearest neighbors in the feature space by an edge.

[0163]

[0164] in This represents the distance between two objects; the smaller the value, the closer they are. (In the adjacency matrix...) Add self-connects and normalize the adjacency matrix.

[0165] , ,

[0166] in for The degree matrix, It is the identity matrix. For each mode... m Using the same set of shared parameters By performing a graph convolution mapping function, the shared embedding representation is obtained as follows:

[0167] ,

[0168] in It is an element-wise nonlinear activation function.

[0169] (2-2) Unique Embedding Representation Extraction

[0170] Will A unique GCN encoder with non-shared input parameters and a unique embedded representation of the output modality. , , and During the training phase, all four modalities are input; during the deployment phase, the external behavioral modality is used. participate.

[0171] Unique graph construction: for each modality , record the unique picture ,in, A unique graph network structure for each mode m, Representing modes The set of nodes below, This represents the edge connections between nodes.

[0172] To the unique graph of the unique adjacency matrix Using independent KNN strategy, the nearest neighbor structure is calculated separately for each modality (no structure is shared between modalities). The adjacency matrix and its normalized form are,

[0173] , ,

[0174] where is the node degree matrix, and the diagonal elements are the number of connections of each node. For each modality , the parameters of the unique encoder are not shared, which is used to capture video, semantic, structural and physiological features, etc. unique to the modality,

[0175] ,

[0176] where is the GCN weight matrix in the unique encoder, which is learned separately for each modality, is a nonlinear activation function.

[0177] (3) Shared-unique embedding representation decoupled fusion mechanism

[0178] The goal of shared-unique embedding representation decoupled fusion is mainly around three goals of orthogonal constraint, embedding representation reconstruction and shared embedding representation alignment to construct a joint optimization loss target.

[0179] (3-1) Shared embedding representation and unique embedding representation orthogonal constraint

[0180] Shared embedding representation characterizes cross-modal general information, and modality unique embedding representation captures modality-specific difference information. Since the shared encoder and the unique encoder may extract redundant or overlapping information, it is easy to cause representation confusion. Therefore, the shared and unique representations of the same modality are subjected to approximate orthogonal constraint, so that they are as linearly independent as possible,

[0181]

[0182] In order to achieve the above constraint, the orthogonal constraint loss function is introduced, which needs to calculate the orthogonal constraint loss of a single modality for each modality,

[0183]

[0184] For the embedding representation of all the above four modalities, the orthogonal constraint loss is,

[0185] .

[0186] where is the Frobenius norm, denotes the sum of squares of all elements. This loss will be incorporated into the model. The above loss is incorporated into the joint optimization objective in the training phase, with the weight coefficient of the orthogonal loss is adjusted between 0.01 and 1.0.

[0187] (3-2) Embedding representation reconstruction

[0188] The purpose of embedding representation reconstruction is to ensure that each modality shares and unique decomposition without losing the original information. For modality , its shared embedding representation and unique embedding representation are spliced into the fusion representation,

[0189]

[0190] For each modality m, a unique reconstruction decoder is constructed respectively The reconstruction decoder is generally composed of a three-layer MLP structure. The spliced embedding representation is input into the modality unique reconstruction decoder to reconstruct the modality input feature ,

[0191] , ,

[0192] where denotes the reconstructed feature of modality N is the number of nodes within the sample, and has the same dimension as the original input .

[0193] In this process, for each modality m, the reconstruction process uses mean square error (MSE) to define the modality embedding representation reconstruction loss,

[0194] ,

[0195] where, is the L2 norm

[0196] The reconstruction losses of all modalities are summed up to form the embedding representation reconstruction loss of embedding representation reconstruction,

[0197] .

[0198] The weight coefficient of the embedding representation reconstruction loss is suggested to be set between 1.0 and 10.0.

[0199] (3-3) Shared embedding representation alignment

[0200] The shared embeddings alignment aims to achieve explicit alignment of shared extrinsic and neural modalities for the same sample within a unified shared space. Cross-modal contrastive learning is performed using fNIRS modalities as teacher anchors. First, the average extrinsic modality vector for each sample is calculated.

[0201]

[0202] in This indicates that the mean value is calculated over the node dimension.

[0203] Secondly, using the fNIRS modality as the teacher anchor for this sample, it is only required that the shared representation of the sample's external modality can be as close as possible to the fNIRS modality of this sample. and with other samples Extend the distance. In a batch (in this invention example set to...) B Within a range of 16, positive sample pairs are formed by the extrinsic mode and its fNIRS mode of the same sample, and negative samples are formed by the fNIRS of the extrinsic mode and other samples in the same batch. Let the similarity be cosine similarity. Temperature coefficient Then the shared embedding alignment loss is,

[0204] ,

[0205] in Indicates the first One sample, modality The sample-level shared vector consists of a numerator corresponding to the positive pairing of the extrinsic and internal fNIRS modalities of the sample within the same contextual task, and a denominator comparing the negative samples after normalization of all intra-batch fNIRS modalities. The shared embedding representation loss is incorporated into the joint optimization during the training phase, with weights denoted as... Its value is recommended to be between 0 and 0.1. It is not calculated during the deployment phase. .

[0206] The adjacency relationships of nodes within each mode of the shared encoder and the unique encoder described above are determined by the K-nearest neighbor algorithm, where... Values ​​range from 5 to 15. The parameters of the shared encoder are shared across the four modalities, while the unique encoders operate independently, learning the differential features within each modality. (Figure: Number of convolutional layers) Set as 1 to 2 layers, with input dimensions as follows: The output dimension is consistent with the input dimension. During the training phase, four modalities are trained. During the deployment phase, only the external modes are calculated. The fNIRS modality is subsequently derived from the reconstructed neural agent features to obtain corresponding components, in order to maintain consistency with the fusion path during training.

[0207] (4) Neural agent feature generation.

[0208] The MFE-CCAGNN model unit needs to complete the external modality and internal neural modality data display alignment in the training stage. In the deployment stage, only the external modality is relied on to complete the discrimination. The multi-modal feature fusion module only receives the external modality in the deployment stage, and the fNIRS branch is replaced by the neural agent feature generated by the external modality.

[0209] In the cross-modal neural agent feature generation stage, the fusion embedding of the input sample external behavior modality (X) v , a , t The reconstruction decoder specific to the fNIRS modality is constructed to reconstruct the approximate representation of , that is, the neural agent feature . The neural agent feature replaces the real fNIRS modality in the deployment stage to participate in subsequent fusion and discrimination.

[0210] The neural agent feature reconstruction is

[0211]

[0212] Among them , and are the fusion embedding of each external modality, is the splicing process. Finally, the fNIRS-specific reconstruction decoder is used for external-to-internal agent reconstruction, and are of the same dimension.

[0213] The neural agent feature reconstruction loss in the above process is obtained by adding the neural agent feature MSE loss and the neural agent feature geometry loss. The neural agent feature MSE loss is defined as

[0214]

[0215] At the same time, it is also necessary to ensure that the reconstructed features and the relative relationship between the nodes is similar, avoiding fitting only on the point value and destroying the geometric structure of the time sequence or segment. The neural agent feature geometry loss is defined as

[0216]

[0217] Among them represents the intra-sample (node dimension) Gram similarity matrix, is the L2 norm.

[0218] The neural agent feature reconstruction loss is

[0219] .

[0220] where and are the neural agent feature MSE loss and neural agent feature geometry loss weight respectively, and the ratio of their values is : ≈1:[0.1-1]. The neural agent feature reconstruction loss needs to be included in the joint optimization loss, and the loss weight is denoted as , which is recommended to be set between 0.5 and 5.0.

[0221] In the deployment stage, there is no longer a need to collect the fNIRS modality, and the generated by the external modality is directly used as the neural agent feature, and then the same fNIRS modality branch encoding and fusion process as in the training stage is used for training to ensure that the inference path is aligned. In this way, the system can be ensured to not rely on the fNIRS modality in the deployment stage, allowing the system to automatically align the data input by the external modality.

[0222] (5) Attention weighted fusion

[0223] In both the training stage and the deployment stage, the attention mechanism needs to be used to assign weights to the shared embedding representation and the unique embedding representation respectively, and the fused final representation is generated, which is input to the downstream classifier module. First, ensure that the shared and unique feature dimensions are consistent, and for each modality , the scalar gating coefficient is calculated, where the gating weight is constructed by a lightweight MLP,

[0224] ,

[0225] where is the sigmoid function. On this basis, the attention weighted fusion formula is constructed as follows,

[0226] .

[0227] In the training stage, the weighted fusion features of the four modalities are spliced to form the final representation,

[0228]

[0229] In the deployment stage, the neural agent obtained in (3-3) is used as the equal-dimensional input of the fNIRS modality, and the neural agent shared embedding representation n and the unique embedding representation of the neural modality are obtained respectively, and then is calculated, replacing the training stage , and finally the weighted fusion feature .

[0230] The classifier module takes the fusion feature as input and outputs the depression classification result. The classifier module is composed of two fully connected layers and a softmax function. Finally, the four-class results are output, which are healthy, mild, moderate, and severe.

[0231] The working process is as follows:

[0232] First, the output of the two fully connected layers in the forward calculation is calculated as

[0233] ,

[0234] The output of the two fully connected layers is calculated using the function to calculate the probability of four depression levels, and finally the class index of depression is output,

[0235]

[0236] where , are the weights of the first and second fully connected layers, respectively, and are the corresponding biases. is a nonlinear activation function. In the final classification stage, let the one-hot vector of the real label be y∈{0,1} (the correct class is 1 and the rest is 0), and the class loss weight , , then the classifier module loss is designed as

[0237] .

[0238] The MFE-CCAGNN model unit needs to be trained before deployment, and the specific settings of the training process are as follows:

[0239] (1) The MFE-CCAGNN model unit needs to be optimized with multiple tasks, and its loss function is set as:

[0240] ,

[0241] where is the classifier module loss, and are the orthogonal constraint loss and its weight, respectively, and are the neural agent feature reconstruction loss and its weight, respectively, and are the embedding representation reconstruction loss and its weight, respectively, and are the shared embedding alignment loss and its weight, respectively. The deployment stage does not participate in the backpropagation and parameter update of any loss term.

[0242] (2) Optimizer and learning rate strategy: In the training stage, the Adam optimizer is used for parameter update, with an initial learning rate of 0.001 and a weight decay term of to suppress overfitting; the StepLR learning rate scheduler is used, multiplying the current learning rate by 0.1 every 10 training cycles to gradually reduce the learning rate and promote model convergence; to reduce the overfitting problem brought by small data sets, 5-fold cross-validation is used for training.

[0243] (3) Training cycle and repeatability control: In the training stage, the total number of training rounds is set to 45, and the training and validation losses are recorded in each training round; to ensure the repeatability of the experimental results, a fixed random seed (seed = 42) is used to control the initialization and data partitioning process.

[0244] (4) Evaluation strategy: In the training stage, the macro-averaged F1-score on the validation set is used as the main evaluation indicator in each fold validation, which is used to select the optimal model parameters; at the same time, Accuracy, Precision, Recall, etc. are recorded for comprehensive performance evaluation. The calculation method is as follows:

[0245] , , , ,

[0246] where TP and TN are defined as the number of correctly classified depressed patients and HC individuals, respectively. The number of healthy individuals classified as depressed and the number of depressed patients classified as healthy represents the false positive (FP) and false negative (FN) values.

[0247] (5) Training environment configuration: The training process is carried out on a single-card GPU (such as NVIDIA RTX 3090), relying on the PyTorch 1.12 deep learning framework, with CUDA version 11.3 and Ubuntu 20.04 operating system.

[0248] System deployment:

[0249] The system of the present application is deployed on an integrated evaluation terminal or a portable psychological screening device with audio and video collection, fNIRS signal access and graphical interaction capabilities, and can be applied to various scenes such as hospital outpatient service, community health service center, school psychological room, etc. On the deployed device, the following methods can be used to display and output the depression recognition results:

[0250] (1) Local terminal display module: on the front-end touch display screen of the device, the individual's depression level judgment result and the corresponding probability information are displayed in a visual graphical manner in real time, which is convenient for on-site rapid evaluation and feedback;

[0251] (2) External interface communication module: through Bluetooth or WiFi module, the recognition results and basic information are pushed to the hospital information system or remote psychological intervention terminal in a structured JSON format, realizing data sharing and remote decision support;

[0252] (3) Local data recording module: the depression level result and its probability distribution of each recognition are written into the local SQLite database, which is used for individual historical state management, psychological intervention effect evaluation and subsequent long-term tracking analysis.

[0253] Verification:

[0254] This embodiment shows the running process of the multi-modal depression recognition system based on the MFE-CCAGNN model of the present application in the real subject task situation, including data collection, stimulus design, model training process and experimental results.

[0255] 1. Data collection and preprocessing

[0256] 1-1 Collection environment and device configuration. The subjects complete the experimental task in a quiet, well-lit, and interference-free environment. In this embodiment, the following devices are used to realize the synchronous collection of multi-modal data:

[0257] (1) Video acquisition module: equipped with a high-definition camera with a frame rate of 30fps, used to obtain facial video data;

[0258] (2) Audio acquisition module: use a high-fidelity microphone with a sampling rate of 44.1kHz;

[0259] (3) fNIRS data acquisition module: use BS-7000 brain function imaging system, covering the prefrontal cortex area, sampling frequency is 20Hz, light source wavelength is 690nm and 830nm.

[0260] Ensure that the audio, video and fNIRS signals are strictly aligned and have a uniform timestamp.

[0261] It also includes a text transcription tool: transcribe audio data into text data.

[0262] 1-2 Data Acquisition Process Steps.

[0263] (1) Subjects wear the fNIRS device and complete position calibration to ensure signal stability;

[0264] (2) Conduct a resting state data acquisition (1 minute) before the task and record the neural background signals;

[0265] (3) Entering the task stage, semantic fluency prompts or emotional pictures are presented, and the subjects express themselves verbally while the trimodal data is recorded.

[0266] (4) Record of resting state after the task (1 minute).

[0267] The multimodal data collected synchronously during the above process will be preprocessed and then input into the video feature extraction module, audio feature extraction module, text feature extraction module and fNIRS data feature extraction module of the present invention system.

[0268] Data was collected from 135 participants, and all data were obtained with informed consent and ethical approval.

[0269] 2. Experimental Tasks and Stimulus Situation Design

[0270] To fully elicit depression-related emotional characteristics, the stimulus materials are divided into the following four scenarios:

[0271] (1) Positive: Open-ended questions that evoke positive emotions;

[0272] (2) Neutral: Daily descriptive tasks, used as a control group;

[0273] (3) Negative: semantic stimuli that induce sadness and depression;

[0274] (4) Resting state: No task, used for comparison with individual baseline control.

[0275] 3. Classification effect in four stimulus scenarios

[0276] In this embodiment, the classification performance of the multimodal depression recognition system based on the MFE-CCAGNN model proposed in this invention was quantitatively verified. The classification results under four scenarios (positive, neutral, negative, and resting state) are as follows: Figure 8 The confusion matrix shown is illustrated below. Each row in the confusion matrix represents the true label category, and each column represents the model's prediction result. There are four categories, each corresponding to a different degree of depression.

[0277] In positive situation, the system performs well on all categories, especially the recognition accuracy of the 2nd (mild) and 4th (severe) categories reaches 90.91% (10 / 11) and 91.67% (11 / 12) respectively, showing good stability and anti-interference ability.

[0278] In neutral situation, the overall classification accuracy remains at a high level except for individual samples with prediction bias between adjacent categories. The severe category (4th) is still accurately classified (11 / 12), indicating that the model has good recognition ability for individuals with severe depression in weak emotional expression situation.

[0279] In negative situation, the system performs particularly outstanding. The classification error of all categories is further reduced. Among them, the accuracy of the 3rd and 4th categories increases to 90.91% and 91.67% respectively, and the first category (no depression) reaches 87.5% (7 / 8), reflecting the model's better discrimination in negative emotional stimulation.

[0280] In resting state situation, the system still maintains stable classification performance, and the performance of each category is basically consistent with that in neutral situation, especially in severe depression samples, which still reaches 91.67% accuracy.

[0281] 4. Performance analysis of four feature extraction modules

[0282] In the deployment state, the performance of the system is compared by inputting three external modalities (video, audio, and text), inputting fNIRS modality alone, and inputting all three external modalities, as shown in Figure 9 The horizontal axis represents the input modality, and the vertical axis represents the performance score. In different stimulus situations (positive, neutral, negative, and resting state), the performance of single modality feature extraction model in depression recognition has obvious differences and limitations. Taking positive situation as an example, the accuracy of video modality is 71.43%, which is better than audio 61.9%, text 64.29%, and fNIRS 57.14%. But in neutral, negative and resting state situations, the accuracy of all single modalities is generally lower than 70%, and the Precision, Recall, F1-Score and other indicators show synchronous decline, indicating that single modality cannot fully capture the overall picture of the subject's emotional state, and there are problems such as insufficient robustness and poor adaptability to specific situations. Further, it shows that in the deployment state, if multiple external modalities are input, the recognition blind spots of single modality in different situations can be effectively compensated, and the recognition ability and stability of the system for complex emotional state can be improved.

[0283] 5. Performance comparison of advanced system

[0284] The embodiment also compares the MFE-CCAGNN model unit in the proposed multi-modal depression recognition system based on the MFE-CCAGNN model. We replace the MFE-CCAGNN model unit with the current mainstream deep learning model under the same conditions of data acquisition unit and data preprocessing unit to verify the effectiveness of its performance. As shown in Figure 10 These deep learning models mainly include ResNet, CNN1D, LSTM, GRU, CNN-LSTM, and CNN-GRU. The comprehensive comparison results under four stimulation situations (positive, neutral, negative, and resting state) show that the MFE-CCAGNN model is significantly better than other comparison models in terms of Accuracy, Precision, Recall, F1-score, and other key evaluation indicators. In the positive situation, the accuracy of the MFE-CCAGNN model is 85.71%, and the F1-score is 0.8511, which is much higher than the ResNet (64.29%) and GRU (73.81%) models. In the neutral situation, the accuracy of the MFE-CCAGNN model is 83.33%, and the F1-score is 0.8294, which is about 14% higher than the CNN-LSTM. In the negative situation, the MFE-CCAGNN model performs the most outstandingly, with an accuracy of 90.48% and an F1-score of 0.9025, which is nearly 20% higher than the ResNet, indicating that the MFE-CCAGNN model has stronger discrimination ability under strong negative emotional stimulation. In the resting state situation, the MFE-CCAGNN model also maintains a leading advantage, with an accuracy of 83.33% and an F1-score of 0.8294, which is better than all other baseline models.

[0285] 6. Deployment complexity and device utility analysis

[0286] In this embodiment, the multi-modal depression recognition system based on the MFE-CCAGNN model is integrated and deployed in a special device including a processor unit, a multi-modal data acquisition interface, and a model inference engine, with the following related parameters:

[0287] (1) The processor unit supports multi-thread parallel operation (Intel i7 platform);

[0288] (2) The system inference engine is composed of the MFE-CCAGNN model unit, which is deployed as ONNX format by PyTorch;

[0289] (3) The modal input interface includes a camera (for video acquisition), a microphone (for audio acquisition), an fNIRS device (for neural imaging acquisition), and a touch screen (for text input and control);

[0290] The deployment performance parameters are as follows:

[0291] (1) Total number of model parameters: 301,601;

[0292] (2) Inference time (single individual data): 2.01 seconds;

[0293] (3) Device-side storage overhead: about 12.3 MB;

[0294] (4) Memory occupation: about 184 MB, suitable for mainstream mobile terminals or embedded platforms.

[0295] The results show that the system has compact structure, low computational resource occupation, and is suitable for deployment on edge devices, notebooks or mobile medical terminals on the basis of ensuring multi-modal fusion performance; the system has fast inference speed and is suitable for large-scale rapid depression state recognition and real-time feedback; the system has good scalability and supports modal combination configuration and module-level update. In summary, the depression recognition device has high performance, low complexity and high adaptability, and is suitable for various application scenarios such as primary screening, clinical auxiliary diagnosis and public mental health intervention.

[0296] The above description is only a description of the preferred embodiments of the present application, and is not any limitation on the scope of the present application. Any modification or modification made by any person skilled in the art according to the technical content disclosed above shall be regarded as an equivalent effective embodiment, and shall fall within the scope of protection of the technical scheme of the present application.

Claims

1. A multi-modal depression recognition system based on an MFE-CCAGNN model, characterized in that, It includes a data acquisition unit, a data preprocessing unit, and an MFE-CCAGNN model unit, wherein: The data acquisition unit is used to simultaneously collect video modal data, audio modal data, text modal data, and fNIRS modal data of the subjects in a standardized interview task scenario during the training phase, and to collect only external behavioral modal data in the same task scenario during the deployment phase. The external behavioral modal includes video, audio, and text. The data preprocessing unit includes a video preprocessing unit, an audio preprocessing unit, a text preprocessing unit, and an fNIRS preprocessing unit; The MFE-CCAGNN model unit includes four feature extraction modules: video feature extraction module, audio feature extraction module, text feature extraction module, and fNIRS data feature extraction module, as well as a multimodal feature fusion module and a classifier module; The data acquisition unit provides time-aligned raw data support for the data preprocessing unit; the inputs of each feature extraction module of the MFE-CCAGNN model unit are connected to the outputs of the corresponding preprocessing modules of the data preprocessing unit to extract features from various types of data output by the data preprocessing unit; the multimodal feature fusion module of the MFE-CCAGNN model unit completes explicit alignment, sharing, and unique embedding decoupling and neural proxy feature generation during the training phase, and generates neural proxy features based solely on external behavior without using neural modalities during the deployment phase, and fuses them with external behavior features, and finally the classifier module outputs the depression recognition result; The multi-modal feature fusion module of the MFE-CCAGNN model unit, input of which is the output of the four feature extraction modules 、 、 、 ; in the training stage, deep embedding mining and explicit alignment of the four types of features are performed, and optimization of cross-modal contrast and latent space reconstruction combined with orthogonal constraints is executed; in the deployment stage, only 、 、 is input, neural agent features corresponding to NFS are generated and fused with external behavior features, and the fused features are output; The construction of the multimodal feature fusion module is based on the mechanism of shared-unique embedding representation decoupling fusion. The shared-unique embedding representation decoupling fusion mechanism refers to applying orthogonal constraints to shared and unique embeddings within the same modality, and combining embedding representation reconstruction and shared space alignment with fNIRS as teacher anchor points, as well as generating neural agent feature reconstruction from the inside out, thereby achieving a method of robust cross-modal fusion. The neural proxy feature generation requires the MFE-CCAGNN model unit to explicitly align external modal and internal neural modal data during the training phase; during the deployment phase, it relies solely on the external modal to complete the discrimination; the multimodal feature fusion module only receives the external modal during the deployment phase, where the fNIRS branch is replaced by the neural proxy features generated from the external modal. The cross-modal neural agent feature generation stage inputs the fusion embedding of the external behavior modal (of the sample) v , a , t ) of the fNIRS modal to construct a reconstruction decoder that reconstructs an approximate representation of , i.e., a neural agent feature ; the neural agent feature replaces the real fNIRS modal in the deployment stage to participate in subsequent fusion and discrimination; The neural agent features are reconstructed as follows: wherein , and are the fusion embedding of each extrinsic modality, is the concatenation process; finally, the fNIRS-specific reconstruction decoder is used to perform the extrinsic-to-intrinsic proxy reconstruction, and are the homed dimensions; fNIRS modality is no longer needed in the deployment phase, directly using the external modality generated as the neural agent features, then using the same fNIRS modality branch encoding and fusion process as in the training phase to ensure the alignment of the inference path; thus ensuring that the system does not rely on the fNIRS modality in the deployment phase, enabling the system to automatically align the data input by the external modality; Multimodal signals in the same task context simultaneously contain shared representations shared with other modalities and unique representations that only appear in this modality.

2. The multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The video preprocessing unit of the data preprocessing unit is used to transform the raw video modal data into a structured, modelable low-level structured descriptor, and finally output video preprocessing data, VPRE. The video preprocessing unit includes a face detection module, a video frame extraction module, and a low-level descriptor extraction module, which are connected in sequence: (1) The face detection module accurately locates the subject's facial area from the input raw video data, removes background and non-facial interference information, and ensures that subsequent processing focuses on the effective area; (2) Video frame extraction module, used to extract frames that can remove redundant frames and unify the face orientation, reduce noise interference caused by pose differences, and perform frame-level sampling; (3) Low-level descriptor extraction module, built using the open-source tool OpenFace, extracts facial key point location, head posture and eye movement data of the subjects during the interview task. These data serve as the biometric features of facial expressions in depression identification. The data output by the low-level descriptor module includes facial expression action unit data, including: Facial motion unit (AUs) intensity: intensity values ​​of 17 motion units; Facial key points: 68 two-dimensional coordinate points, totaling 136 dimensions; Head pose: X / Y / Z 3D position and Pitch / Yaw / Roll rotation angles, totaling 6 dimensions; Eye movement direction: a three-dimensional vector of the left and right eye gaze directions, totaling 6 dimensions; The final output VPRE of the video preprocessing unit is a total of 165 dimensions; The video feature extraction module of the MFE-CCAGNN model unit, whose input is the video coding feature VPRE output by the video preprocessing unit, and whose output is Let the frame sequence of VPRE be denoted as where each frame The video feature extraction module constructs an emotion activity gating-Transformer structure, which includes an emotion score calculation block, an emotion activity gating weighting block, and a Transformer encoder. The Transformer encoder structure is stacked with two layers, each layer containing four multi-head attention heads, each attention head with a dimension of 32, a total input dimension of 165, and a feed-forward network (FNN) with a dimension of 512. All layers employ residual connections and LayerNorm normalization mechanisms. Its working process is as follows: (1) Calculate the input features and emotion scores using the emotion score calculation block; each time video frame in the VPRE data representing the structured video behavior features of the frame, first compute an emotion score for each frame and compress to the (0, 1) interval, wherein denotes a Sigmoid function; (2) Perform emotional activity gating weighting operation through emotional activity gating weighting block; A gating structure is used to fuse frame features guided by sentiment weights with the original features. ， wherein , , is the emotion score, are the pre- and post-weighted frame feature representations, respectively, and ReLU(·) is a non-linear activation function. (3) Construct the Transformer encoder; (3.1) Firstly, sequence construction and linear mapping are performed: all frames are constructed into a new video sequence, and the embedding dimension is converted using linear mapping to adapt the Transformer structure, and the input dimension is kept as 165; (3.2) Secondly, multi-head attention modeling of inter-frame dynamic association is performed: a stacked multi-head self-attention mechanism is used to perform context modeling of the emotion-weighted frame sequence. , wherein is a stacked computation, , is an output mapping matrix, and ， The above Q, K, V are respectively Query, Key, Value vector sequences, is a multi-classification function; (3.3) Finally, nonlinear transformation and residual connection are performed: the output sequence is transformed through a feedforward network and residual connection to obtain the output of the video feature extraction module. , wherein represents a feedforward network, is layer normalization, represents an output of the video feature extraction module.

3. The multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The audio preprocessing unit of the data preprocessing unit processes the raw audio speech data and finally outputs audio preprocessing data, APRE. The audio preprocessing unit includes a filtering and noise reduction module, an amplitude normalization module, and a silence data removal module, which are connected in sequence as follows: (1) Filtering and noise reduction module: This module uses a Butterworth bandpass filter to perform frequency domain filtering on the original audio data, filtering out environmental interference signals that are too high or too low, retaining the human voice frequency band in the range of [80Hz, 8000Hz], and filtering out low-frequency noise, current interference and equipment noise; (2) Amplitude normalization module. The function of this module is to perform amplitude normalization processing on the denoised audio data in order to eliminate the influence of volume differences between different individuals or recording environments. (3) Silence data removal module: This module removes silence segments from the normalized audio data to eliminate invalid pauses and background noise interference, retaining valid speech segments. Specifically, the silence data removal module of the audio preprocessing unit removes silence segments using the frame energy determination method, as shown below: , wherein, is the frame energy, is the input speech data, represents a speech segment, represents the total number of speech segments; if a frame energy satisfies wherein is a threshold of silence, the frame is identified as an invalid silence frame and is eliminated; (4) Mel power spectrum calculation module. This module is used to calculate the Mel spectrum of the audio data after removing silence. Specifically, this module uses short-time Fourier transform to calculate the power spectrum and uses the Mel filter bank to map to the sensing frequency space to generate a 128-dimensional Mel frequency energy spectrum. The calculation formula is as follows. ， wherein is the audio frequency of the speech data, is converted from ; is the corresponding Mel frequency, i.e. the APRE output by the audio pre-processing unit; The audio feature extraction module of the MFE-CCAGNN model unit, the input of which is the APRE output by the audio preprocessing unit, and the output is the audio coding feature ; In the audio feature extraction module, the multi-scale convolutional structure contains three one-dimensional convolutional kernels with sizes of 1×1, 1×3, and 1×5, respectively. Each kernel has 64 channels, and the total number of output channels after concatenation is 192. The convolutional layer is followed by BatchNormalization and ReLU activation functions. To ensure alignment with the original spectrogram in the time dimension, the convolution uses a stride of 1 and symmetrical padding to maintain the same time length. Its working process is as follows: (1) First, for the input APRE data, 1×1, 1×3, and 1×5 convolutional kernels are constructed to extract local details, short-term rhythm, and sliding frequency band changes, respectively. The outputs of the three are concatenated to form a feature fusion representation. , wherein an input APRE representing audio data, represents a one-dimensional convolution along the time dimension, each branch output channel number is 64, concatenate by channel dimension, k takes 1, 3, 5; (2) Channel normalization and activation; On fusion features BatchNorm and ReLU operations are performed to improve the network's nonlinear modeling ability and numerical stability. ， wherein, For the post-activation feature, the time length remains the same, and the number of channels is 192. (3) Channel residual fusion; Activated features With the original input After addition, the final output of the audio feature extraction module is: .

4. The multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The text preprocessing unit of the data preprocessing unit preprocesses the acquired raw text data and finally outputs text preprocessing data, TPRE; the text preprocessing unit performs data cleaning and standardization. The text feature extraction module of the MFE-CCAGNN model unit takes the TPRE output by the text preprocessing unit as input and outputs the text encoded features as output. ; The text data feature extraction module constructs an N-Transformer structure, which includes multiple Transformer blocks and uses a stacked multi-head attention structure to model the contextual relationships between words. Emotional region attention is introduced in the embedding layer. In the text feature extraction module, the maximum length of the input text sequence is 128. First, the word vectors are projected to a dimension of [dimensional value missing] through the embedding layer. d In a space of 256, a text embedding matrix is ​​formed; the subsequent input is fed into an 8-layer Transformer structure, each Transformer layer contains 8 attention heads, each attention head has a dimension of 32, and the intermediate dimension of the feedforward network is 512. Its working process: (1) Embedding layer and sequence representation; The sequence of TPRE outputs from the text preprocessing unit is represented as follows: ， ， in Represented as the first i The maximum sequence length of the words is d =128; the word vectors are mapped to a dimension of through the embedding layer. From the embedding space, we obtain the embedding matrix. , represented as: in It is the embedding layer, which integrates word or sub-word vectorization with positional encoding; (2) Multi-head self-attention and emotional region attention mask; Embedding matrix The input is fed into a multi-layered stacked Transformer module, each layer containing a multi-head self-attention module and a feedforward network module. The standard multi-head self-attention calculation is as follows: . Q, K, and V are obtained by linear mapping from the input embedding matrix. The dimension is single-head key vector; a region-of-interest attention mechanism based on grammatical error tags is constructed using a mask. in This represents the original attention distribution. This represents the attention area mask constructed from grammatical error tags; 1 indicates retention, and 0 indicates masking. The output for each attention is: Finally, we obtain the context feature representation matrix after Transformer encoding. (3) [CLS] Convergence and Dual-Channel Output; From the context feature matrix Extract the [CLS] flag bit vector Using dual-channel output, including channel-based computation of contextual features, a sentiment semantic representation is obtained. ; Language consistency score of context features Finally, the text features are merged. ， in This represents the output of the text feature extraction module. This represents the sigmoid function.

5. A multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The fNIRS preprocessing unit of the data preprocessing unit processes the fNIRS signals collected from the subjects during the interview task or cognitive stimulation process, and finally outputs fNIRS pre-processing data, NPRE. Specifically, the fNIRS preprocessing unit includes an HbO concentration estimation module, a bandpass filtering and noise reduction module, and an HbO signal baseline correction module, which are connected in sequence as follows: (1) HbO concentration estimation module. This module first performs a logarithmic transformation on the fNIRS light intensity data to obtain the change in optical density, and then uses the modified Bell-Lambert law to estimate the change in hemoglobin concentration. ; (2) Bandpass filtering and noise reduction module. The function of this module is to measure the change in the original hemoglobin concentration. Bandwidth restriction is applied to filter out non-cognitive components; (3) HbO signal baseline correction module, which measures the change in hemoglobin concentration in each channel. Baseline correction is performed by shifting the signal to remove baseline deviation during the resting period. The specific calculation method is as follows. ， in, This represents the change in hemoglobin concentration in a channel at time t before baseline correction. , Represented as baseline corrected , Indicates the duration of the resting period, used to estimate the baseline mean; It is an integral variable; The fNIRS feature extraction module of the MFE-CCAGNN model unit takes the NPRE output by the fNIRS preprocessing unit as input and outputs the fNIRS encoded features. ; The fNIRS feature extraction module first calculates the Pearson correlation matrix between different channels to describe the neural collaboration pattern under the task state. Then, to address the problems of high-dimensional redundancy, low-frequency drift and complex connection patterns caused by multi-channel acquisition, it adopts two-dimensional convolution and multi-path pooling connection. The multi-path pooling includes max pooling, average pooling and attention pooling. This module is used for mapping learning and explicit alignment during the training phase, and does not call fNIRS during the deployment phase. The fNIRS feature extraction module of the MFE-CCAGNN model unit includes a Pearson correlation matrix calculation module, a two-dimensional convolutional block, and a multi-path pooling block. Specifically, the input is a 53×53 correlation matrix, the convolutional kernel size is 3×3, the number of channels is 64, and the multi-path pooling block uses three pooling paths, namely max pooling, average pooling, and attention pooling. Its workflow is as follows: (1) The Pearson correlation coefficient of HbO signals between all channels was calculated using the Pearson correlation matrix calculation module; a 53×53 connectivity matrix was output, which represents the co-activation pattern of different brain regions during the subject's task execution, characterizing the individual's neuromodulation features in a depressive state. ， in and This is a channel i and j HbO signal; (2) Use convolution kernels to extract local neural connectivity patterns. (3) The convolutional features are subjected to three parallel pooling methods: max pooling to capture high-intensity co-activation regions, average pooling to characterize the overall connectivity stability, and attention pooling to improve the model's ability to focus on key connectivity pathways. Among these, max pooling features... F fnirs-max Average pooling characteristics F fnirs-mean Attention pooling features F fnirs-attn The calculations are as follows: , , ， in It is the importance weight of the attention pooling learning region. The three types of pooling features are concatenated along the channel dimension to form a multi-view neural connectivity graph representation. 。 6. A multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The multimodal feature fusion module of the MFE-CCAGNN model unit takes the outputs of the four feature extraction modules as its input. , , , During the training phase, deep embedding mining and explicit alignment are performed on the four types of features, and optimization is carried out using joint orthogonal constraints of cross-modal contrastive and latent space reconstruction; during the deployment phase, only... , , As input, generate neural agent features corresponding to NFS and fuse them with external behavioral features, outputting fused features. ; The construction of the multimodal feature fusion module is based on the mechanism of shared-unique embedding representation decoupling fusion. The shared-unique embedding representation decoupling fusion mechanism refers to applying orthogonal constraints to shared and unique embeddings within the same modality, and combining embedding representation reconstruction and shared space alignment with fNIRS as teacher anchor points, as well as generating neural agent feature reconstruction from the inside out, thereby achieving a method of robust cross-modal fusion. The workflow is as follows: (1) Unified representation of dimensions; To perform explicit alignment and subsequent fusion within the shared space, the four modal features VFS, AFS, TFS, and NFS are first uniformly mapped to the same dimension. This process uses a two-layer MLP to perform dimension alignment on each modality, resulting in an equal-dimensional representation: ， ， in middle v , a , t , n These represent video (VFS, v ), audio (AFS, a ), text (TFS, t ) and fNIRS (NFS, n Indexes of four modal features It is an element-wise nonlinear activation function. and These are the weight matrix and bias vector of the first-layer MLP, respectively. and These are the weight matrix and bias vector of the second-layer MLP; the final output is the modality. Node feature matrix Correspondingly, , , , These are the equal-dimensional representations of video, audio, text, and fNIRS modalities, respectively. (2) Embedded representation extraction; (2.1) Extraction of shared embedding representations; Will In a GCN shared encoder with shared input parameters, the output is a shared embedding representation of four modalities. , , and During the training phase, all four modalities are input, and during the deployment phase, the external behavioral modality is used. participate; Shared graph construction: for each modality The image is recorded as follows The set of nodes is , Representing modes m The Input feature representation of each node; edge set The K-Nearest Neighbors (KNN) algorithm is used to measure the Euclidean distance of each node in the semantic space, and each node is connected to its K nearest neighbors in the feature space by an edge. ， in This represents the distance between two objects; the smaller the value, the closer they are. (In the adjacency matrix...) Add self-connects and normalize the adjacency matrix. ， ， in for The degree matrix, It is the identity matrix; for each mode m Using the same set of shared parameters By performing a graph convolution mapping function, the shared embedding representation is obtained as follows: ， in Degree matrix The inverse square root, It is an element-wise nonlinear activation function; (2.2) Unique Embedding Representation Extraction Will A unique GCN encoder with non-shared input parameters and a unique embedded representation of the output modality. , , and During the training phase, all four modalities are input, and during the deployment phase, the external behavioral modality is used. participate; Unique graph construction: for each modality , record the unique picture ,in, A unique graph network structure for each mode m, Representing modes The set of nodes below, Indicates the edge connection relationship between nodes; For unique images Unique adjacency matrix Using an independent KNN strategy, the nearest neighbor structure is calculated separately for each modality. The adjacency matrix and its normalized form are as follows: ， ， in Let the node degree matrix be... It is the identity matrix; for each mode The parameters of the unique encoder are not shared; they are used to capture video, semantic, structural, and physiological features specific to each modality. ； in The GCN weight matrix is ​​learned separately for each modality in a unique encoder. It is a non-linear activation function; (3) Shared-unique embedded representation decoupling and fusion mechanism The shared-unique embedding representation decoupling and fusion mechanism aims to construct a joint optimization loss objective around three objectives: orthogonal constraints, embedding representation reconstruction, and shared embedding representation alignment. This objective includes: (3.1) Orthogonal constraints of shared and unique embedding representations Shared Embedded Representation Representing cross-modal general information, modality-unique embedding representation Capture modality-specific differences; impose approximately orthogonal constraints on shared and unique representations of the same modality, making them as linearly independent as possible. ； (3.2) Embedded Representation Reconstruction Embedded representation reconstruction ensures sharing across modalities and unique decomposition Without losing original information; first, for modal Its shared embedded representation and unique embedding representation splicing is represented as fusion. Secondly, a unique reconstruction decoder is constructed for each modality m. The concatenated embedded representation of the input modality is used to uniquely reconstruct the decoder and remodel the modal input features. , ， ， in Representing modes The reconstructed features, where N is the number of nodes in the sample, and is related to the original input. Same dimension; final calculation and The MSE loss is minimized. (3.3) Shared embedding representation alignment Shared embeddings represent alignment that requires explicit alignment of shared extrinsic and neural modalities for the same sample within a unified shared space. Cross-modal contrastive learning is performed using the fNIRS modality as the teacher anchor. First, the average value of the extrinsic modality vector for each sample is calculated. in This indicates that the mean is calculated over the node dimension; secondly, the fNIRS mode is used as the teacher anchor for this sample, requiring only that the shared representation of the sample's external modes be as close as possible to the fNIRS mode of this sample. and with other samples To minimize the impact of fNIRS on the external modality of a sample within a batch, positive sample pairs are formed by combining the external modality with the fNIRS of other samples in the same batch, and negative samples are formed by combining the external modality with the fNIRS of other samples in the same batch. A shared embedding alignment loss is constructed to minimize this loss. The adjacency relationships of nodes within each mode of the shared encoder and the unique encoder described above are determined by the K-nearest neighbor algorithm, where... Values ​​range from 5 to 15; the parameters of the shared encoder are shared across the four modalities, while the unique encoders operate independently, learning the differential features within each modality; the number of convolutional layers is shown in the figure. Set as 1 to 2 layers, with input dimensions as follows: The output dimension is consistent with the input dimension; (4) Attention-weighted fusion Weights are assigned to shared and unique embedding representations for each modality. Calculate the gating coefficient of the scalar The gating weights are constructed using a lightweight MLP. ， in It is the sigmoid function; the attention-weighted fusion formula is, ， During the training phase, the weighted fusion features of the four modalities are concatenated to form the final representation. During the deployment phase, the neural agent obtained in step (3.3) is used. As isodimensional inputs to the fNIRS modality, neural modalities are obtained respectively. n Neural agent shared embedding representation and unique embedding representation Then calculate Replace the training phase Finally, the weighted fusion features are obtained. .

7. A multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The classifier module of the MFE-CCAGNN model unit uses the fused features from the multimodal feature fusion module. The input is depression classification results; the classifier module is composed of two fully connected layers and a softmax function, and finally outputs four classification results, namely healthy, mild, moderate and severe. Its working process is as follows: First, calculate the output of the two fully connected layers in the forward computation. , in , These are the weights formed by the fully connected layers 1 and 2, respectively. and This is the corresponding bias; It is a non-linear activation function; The output of the two fully connected layers is used The function calculates the probability of four possible levels of depression and finally outputs the index of the depression category. In the final classification stage, let the one-hot vector of the true label be y∈{0,1}. The label is for the k-th class, with 1 for the correct class and 0 for the rest. The class loss weights are... , Input the outputs of the two fully connected layers mentioned above. The classification loss is designed as follows: 。 8. A multimodal depression recognition system based on the MFE-CCAGNN model as described in claim 1, characterized in that, The MFE-CCAGNN model unit needs to be trained before deployment. The training aims to minimize the joint loss function, which is: ， in It is the loss of the classifier module. and These are the orthogonal constraint loss and its weight. and These are the neural agent feature reconstruction loss and its weights. and These are the embedding representation reconstruction loss and its weights. and These are the shared embedding alignment loss and its weights; Specifically, orthogonal constraint loss for, ， in , It is the Frobenius norm. Represents the sum of squares of all elements; weighting coefficients Adjust between 0.01 and 1.0; Specifically, embedding represents the reconstruction loss. for, ， For each mode m, the reconstruction process uses mean squared error (MSE) to define the mode reconstruction loss. ， in, For modality The reconstruction features, It is the L2 norm; weighting coefficient Its value is set between 1.0 and 10.0; Specifically, shared embedding alignment loss Using contrast loss, ， in This is the number of samples in a training batch. Indicates the first One sample, modality Sample-level shared vectors, weight coefficients Set between 0 and 0.1; Specifically, neural agent feature reconstruction loss for, ， in and These are the weights of the neural surrogate feature MSE loss and the neural surrogate feature geometric loss, respectively, and their values ​​are in proportion to... : ≈1:[0.1-1]; Neural agent feature MSE loss for, ， in Neural proxy features of fNIRS modality It is a dimensional unified representation of fNIRS modes; The geometric loss weights for neural agent features are defined as follows: ； in It is a Gram similarity matrix; Specifically, in the final classification stage, let the one-hot vector of the true label be y∈{0,1}. Let be the category label of the k-th class, with 1 for the correct class and 0 for the rest. Class loss weights. , Given the output of two fully connected layers, the loss function of the classifier module is designed as follows: 。

Images