Multimodal feature adaptive fusion early detection device for depression

Abstract

This invention discloses a multimodal feature adaptive fusion early detection device for depression, belonging to the field of disease diagnosis based on artificial intelligence technology. The device includes: a data acquisition module to acquire multimodal depression-related data; a feature extraction module to extract deep features of each modality; an adaptive fusion module, including a multi-scale channel attention enhancement unit, a dynamic weight calculation unit, and an attention feature fusion unit, which performs multi-scale channel attention enhancement on the deep features, calculates dynamic fusion weights based on the modality correlation matrix, and performs attention feature fusion; and a classification output module to output depression risk level and abnormal modality source information. This invention significantly improves detection accuracy and robustness by replacing static fusion with an adaptive fusion architecture, is optimized for the physiological characteristics of adolescents, supports lightweight modeling and cross-scenario deployment, and can be widely applied to auxiliary clinical diagnosis in hospitals, large-scale screening in schools, and home monitoring.

Description

Technical Field

[0001] This invention relates to the field of disease diagnosis based on artificial intelligence technology, specifically to an early detection device for depression based on multimodal feature adaptive fusion. Background Technology

[0002] Depression, a common mental disorder with high incidence, recurrence, and disability rates, has become a major health concern in the global public health field. With the accelerating pace of modern life and increasing psychological stress, the incidence of depression is showing a significant trend towards younger ages, with the proportion of adolescent patients rising year by year. Early identification, accurate diagnosis, and timely intervention are crucial for improving patient prognosis and reducing the social burden of the disease.

[0003] Currently, clinical diagnosis and screening for depression mainly rely on psychological scale assessments, clinician interviews, and behavioral observations. Standardized assessment tools such as the Patient Health Questionnaire-9 (PHQ-9), the Hamilton Depression Rating Scale-17 (HDRS-17), and the Beck Depression Inventory (BDI) are widely used and can reflect the degree of depression and emotional state to some extent, providing a basis for clinical diagnosis. However, these methods, which are primarily subjective, are highly dependent on physician experience and patient cooperation. They have limited ability to identify early and mild depressive states and are difficult to objectively quantify, leading to missed diagnoses and misdiagnoses.

[0004] With the rapid development of artificial intelligence, biomedical engineering, and pattern recognition technologies, objective detection methods based on physiological signals and behavioral characteristics are gradually being applied to the auxiliary assessment of depression. Electrocardiogram (ECG) signals can reflect the imbalance between sympathetic and parasympathetic nerve functions in a depressive state; electroencephalogram (EEG) signals, through spectrum, functional connectivity, and entropy analysis, can identify abnormal patterns in specific frequency bands and brain regions of depressed individuals; and speech and facial expression analysis can achieve non-contact emotional state recognition by extracting features such as speech rate, fundamental frequency, and facial action units (AUs).

[0005] To further improve the comprehensiveness and reliability of detection, multimodal fusion detection has become a research hotspot in this field. Existing technologies typically first extract features and perform preliminary modeling for each individual modality, and then integrate multi-source information through feature-level concatenation, model-level ensemble, or decision-level voting. Relying on deep learning models such as convolutional neural networks and long short-term memory (LSTM) networks, effective identification of depressive states has been achieved in laboratory datasets and small-scale clinical samples, driving the evolution of detection technology from subjective assessment to objective quantification, and from single-modality to multimodal fusion.

[0006] However, existing multimodal fusion detection schemes still have many technical shortcomings, making it difficult to meet the needs of clinical and large-scale screening. The inventors independently discovered through research that existing technologies generally employ fusion methods such as independent extraction of single modalities, fixed-weighted fusion, or simple feature splicing. These methods rely on static fusion rules and general models, failing to dynamically adapt to modal characteristics and population differences, resulting in insufficient detection robustness. Fixed weights cannot adjust modal contributions in real time based on scene, individual, and emotional state, easily leading to weakening of key features and amplification of noise information. Simple splicing fails to establish a dynamic interaction mechanism between modalities, making it difficult to uncover deep correlations between physiological and behavioral signals, resulting in high feature redundancy and low utilization. Furthermore, existing models are mostly trained on adult datasets; when directly transferred to adolescents, they cannot adapt to their unstable EEG rhythms and differences in speech and facial expression development, significantly reducing generalization performance. Simultaneously, the complex model structure and large number of parameters, lacking lightweight design, make it difficult to achieve low-latency real-time detection on wearable devices and mobile terminals. Moreover, differences in data distribution across different devices and scenes further lead to insufficient cross-scene generalization ability, affecting the stability of detection results.

[0007] Therefore, how to solve the problems of insufficient robustness and low feature utilization of existing multimodal fusion detection technologies, while improving the model's adaptability to adolescents and its ability to be deployed across scenarios, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] To alleviate or partially alleviate the above-mentioned technical problems, the solution of the present invention is as follows:

[0009] On one hand, this invention discloses a device for early detection of depression based on adaptive fusion of multimodal features, comprising:

[0010] The data acquisition module is used to acquire depression-related data from at least two different modalities of the participants;

[0011] The feature extraction module is used to extract depth features for each modality;

[0012] An adaptive fusion module is used to perform adaptive fusion on the depth features of each modality. The adaptive fusion module specifically includes:

[0013] The multi-scale channel attention enhancement unit is used to perform multi-scale channel attention enhancement on the depth features of each modality to obtain the enhanced single-modality features;

[0014] The dynamic weight calculation unit is used to construct a modality correlation matrix that reflects the pairwise correlation between each enhanced single-modal feature. Based on the sum of the correlation strength between each modality and all other modalities in the modality correlation matrix, the dynamic fusion weight of each modality is calculated.

[0015] The attention feature fusion unit is used to perform weighted fusion of each enhanced single-modal feature according to the dynamic fusion weight, and perform attention feature fusion with the element-wise summation result of each enhanced single-modal feature to obtain a fused feature vector;

[0016] The classification output module is used to output the depression risk level based on the fused feature vector.

[0017] In one embodiment, the multi-scale channel attention enhancement unit is configured as follows:

[0018] The input single-modal features are processed in two parallel paths:

[0019] The first path first performs global average pooling, and then sequentially passes through a bottleneck structure consisting of the first pointwise convolution, the ReLU activation function, and the second pointwise convolution to extract global channel features;

[0020] The second path directly extracts local channel features through a bottleneck structure composed of pointwise convolution, ReLU activation function, and pointwise convolution with the same structure.

[0021] The global channel features and local channel features are added element by element, and then channel attention weights are generated by applying a Sigmoid activation function.

[0022] The channel attention weights are multiplied element-wise with the input unimodal features to output the enhanced unimodal features.

[0023] In one embodiment, the dynamic weight calculation unit is configured as follows:

[0024] The correlation strength between any two enhanced single-modal features is calculated using the Pearson correlation coefficient, and a modal correlation matrix R is constructed. Each element R(m, n) of the modal correlation matrix R is the Pearson correlation coefficient between the m-th mode and the n-th mode, where m and n are the mode numbers.

[0025] When there are 5 modes, the basic contribution weight W of each mode is calculated according to the following formula. m :

[0026]

[0027] Where |R(m, n)| represents the absolute value of the correlation coefficient R(m, n); and,

[0028] The sum of the base contribution weights for all modalities is 1.

[0029] In one embodiment, the dynamic weight calculation unit is further configured as follows:

[0030] The basic contribution weights are adaptively adjusted based on the detection scenario as follows:

[0031] When the detection scenario is a resting state, the weights of the ECG mode and / or EEG mode are multiplied by a factor of 1.5.

[0032] When the detection scenario is task-oriented, the weights of the speech modality and / or facial expression modality are multiplied by a factor of 1.5.

[0033] The base contribution weights of all modalities after correction are normalized so that the sum of the base contribution weights of all modalities is 1, thus obtaining the final dynamic fusion weights of each modality.

[0034] In one embodiment, the attention feature fusion unit is configured as follows:

[0035] The initial aggregated feature F is obtained by summing the elements of each enhanced single-modal feature. agg ;

[0036] The initial aggregation feature F agg Input a multi-scale channel attention unit to generate a global fusion weight M, wherein the value of the global fusion weight M ranges from 0 to 1;

[0037] The enhanced single-modal features are summed element-wise according to the dynamic fusion weights of each modality to obtain the weighted aggregated features;

[0038] The final fused feature vector Z is output using the following formula:

[0039]

[0040] in, This indicates element-wise summation; This represents element-wise multiplication; W1, W2, W3, W4, and W5 are the dynamic fusion weights for each modality; F'1, F'2, F'3, F'4, and F'5 are the enhanced single-modal features; F agg M represents the initial aggregation feature; M represents the global fusion weight.

[0041] In one type of embodiment, the at least two different modes are selected from at least two of the following five categories:

[0042] Scale data, electrocardiogram signals, electroencephalogram signals, voice signals, or facial expression videos.

[0043] In one embodiment, the extraction of depth features for each modality specifically includes:

[0044] For the scale text data, a Transformer encoder is used to extract semantic feature vectors containing sentiment tendencies;

[0045] For electrocardiogram (ECG) and electroencephalogram (EEG) signals, a dual-branch structure of long short-term memory (LSTM) network is used to extract the shared temporal variation features of the two types of physiological signals and the unique physiological features of each modality.

[0046] For speech signals, a convolutional neural network is used to extract deep acoustic features from the spectrogram;

[0047] For facial expression videos, a convolutional neural network with embedded channel attention mechanism and position encoding is used to extract micro-expression features of facial action units.

[0048] In one embodiment, the classification output module is further configured to:

[0049] The dynamic fusion weights of each modality are output synchronously as abnormal modality source information of the detection results, and the key abnormal modalities with the highest contribution are marked.

[0050] In one type of embodiment, it further includes:

[0051] The lightweight deployment module is used to compress the trained model using at least one of the techniques of knowledge distillation, model quantization, and structured pruning; and eliminates data distribution offset between different acquisition devices and detection scenarios through Z-score normalization preprocessing and domain adaptation techniques.

[0052] In one type of embodiment, it further includes:

[0053] The visualization and interactive module is used to display the original data of each modality, feature analysis results, dynamic fusion weights of each modality, and the final depression risk level in real time.

[0054] The technical solution of this invention has one or more of the following beneficial technical effects:

[0055] (1) Significantly improve detection accuracy and robustness. Single-modal features are enhanced by multi-scale channel attention to suppress noise. Then, the fusion weights are dynamically allocated through the modality correlation matrix. Finally, the weighted features and unweighted aggregated features are fused by attention feature fusion to achieve adaptive interpolation fusion, which improves the overall accuracy of early detection of depression, effectively reduces the rate of missed diagnosis and misdiagnosis, and enhances the ability to identify early mild depression.

[0056] (2) It realizes efficient utilization and interpretable detection of multimodal information. By dynamically allocating weights through the modality correlation matrix, the feature utilization rate is improved. At the same time, it can output the contribution of each modality and the source of abnormality, providing an objective and traceable basis for clinical diagnosis.

[0057] (3) It is suitable for adolescents and supports large-scale deployment. It has designed a special feature extraction network for adolescents with unstable brainwave rhythm and fluctuating speech fundamental frequency. The detection accuracy is improved compared with the general adult model. The model is lightweight and suitable for deployment in multiple scenarios.

[0058] Furthermore, other beneficial effects of the present invention will be mentioned in the specific embodiments. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the device composition and data processing flow of one embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of data acquisition according to one embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of the multi-scale channel attention (MS-CAM) unit structure of the present invention;

[0062] Figure 4 This is a schematic diagram of the attention feature fusion (AFF) unit structure of the present invention;

[0063] Figure 5 This is a schematic diagram of the device frame of one embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0065] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order.

[0066] This invention discloses a device for early detection of depression based on adaptive fusion of multimodal features. The device includes a data acquisition module, a feature extraction module, an adaptive fusion module, a classification output module, and optional lightweight deployment and visualization interaction modules. It aims to address the core technical problems in existing multimodal depression detection technologies, such as insufficient robustness due to static fusion rules, low feature utilization, poor adaptability to adolescents, and difficulty in large-scale deployment across various scenarios.

[0067] This invention constructs an adaptive fusion architecture integrating Multi-Scale Channel Attention (MS-CAM), Modality Relevance Matrix, and Attention Feature Fusion (AFF) to replace the traditional fixed-weight weighting and simple feature concatenation fusion method, achieving dual dynamic adaptation at both the sample and scene levels. Simultaneously, it optimizes the feature extraction path to address the characteristics of unstable EEG rhythms and differences in speech and facial expression development in adolescents, and lightweights the model. Ultimately, it constitutes a high-precision, low-latency, highly interpretable, and cross-scenario universal early detection device for depression, which can be widely applied in hospital clinical auxiliary diagnosis, large-scale mental health screening in schools, and home monitoring, especially suitable for identifying depressive states in adolescents.

[0068] Figure 1 This is a schematic diagram of the device composition and data processing flow of one embodiment of the present invention, demonstrating the complete technical chain from multimodal raw data acquisition, preprocessing, feature extraction, adaptive fusion to depression state classification output. As shown in the figure, the device sequentially includes four stages: multimodal data acquisition and preprocessing, deep feature extraction of each single modality, adaptive fusion of multimodal features, and depression state classification and early detection, which are respectively implemented by the data acquisition module, feature extraction module, adaptive fusion module, and classification output module. One embodiment also includes a lightweight deployment module. Among them, the adaptive fusion of multimodal features is the core inventive point of the present invention. It achieves dual adaptive dynamic fusion at the sample level and scene level through a dedicated fusion architecture composed of a multi-scale channel attention (MS-CAM) unit, a modality correlation matrix calculation unit, and an attention feature fusion (AFF) unit, overcoming the problems of key feature weakening and information redundancy caused by traditional fixed weight or simple splicing fusion. The specific implementation methods of each stage are described in detail below.

[0069] Multimodal data acquisition and preprocessing: This step is a necessary prerequisite for providing standardized multi-source basic data for all subsequent processing steps. Figure 2 This is a schematic diagram of data acquisition according to one embodiment of the present invention, showing the layout of the acquisition equipment for the subject's testing posture and various modal data. As shown in the figure, the present invention simultaneously acquires five heterogeneous modal data: scales, electrocardiograms, electroencephalograms, speech, and facial expressions.

[0070] In one embodiment, the subject sits upright at a testing table, with a portable EEG acquisition device worn on the forehead to collect frontal EEG signals, disposable ECG electrodes attached to the chest to collect ECG signals, and a smart terminal placed directly in front of the subject. The screen displays a PHQ-9 questionnaire for the subject to complete, and the terminal's built-in microphone simultaneously collects speech signals. A camera is placed next to the terminal to capture facial expression video. Specific data acquisition parameters are set as follows: ECG signal sampling rate 250Hz, EEG signal sampling rate 500Hz, continuous acquisition for 5 minutes in a resting state; speech signal sampling rate 44.1kHz, quantization precision 24bit, using a standardized text reading task for acquisition; facial expression video resolution 640×480, frame rate 30fps, simultaneously extracting real-time intensity values ​​of 17 facial motion units.

[0071] After acquiring the raw data, targeted preprocessing was performed on different modalities to eliminate noise and artifacts. For ECG signals, wavelet transform was used to remove power line interference and electromyographic noise; for EEG signals, Independent Component Analysis (ICA) was used to remove electrooculography (EOG) artifacts; for speech signals, pre-emphasis, framing, and windowing were performed sequentially, followed by Fast Fourier Transform (FFT) to convert them into a spectrogram reflecting time-frequency characteristics; for facial expression data, head posture correction was performed using Kinect 3D coordinates to eliminate the impact of posture deviation on the accuracy of facial motion unit extraction. Finally, zero-mean (Z-score) standardization was uniformly applied to the feature data of all modalities, mapping each modal data to a standard distribution with a mean of 0 and a variance of 1, to eliminate data distribution offsets introduced by different acquisition devices and usage scenarios, providing a standardized input representation for the model.

[0072] In one embodiment, modalities such as eye movement signals and skin conductance signals can be added to increase feature dimensions and further improve detection accuracy.

[0073] Each single-modal deep feature extraction stage: After preprocessing, for the signal characteristics and semantic features of the five modalities of scale, ECG, EEG, speech, and facial expression, a dedicated deep learning network is constructed to extract features, and the heterogeneous original signals are uniformly transformed into high-level semantic feature vectors of fixed dimensions, providing standardized input for subsequent fusion.

[0074] For the scale text data, the Transformer encoder was used to semantically encode the text responses of the PHQ-9 and BDI scales, and semantic feature vectors containing expressions of emotional tendencies and depressive symptoms were extracted.

[0075] For electrocardiogram (ECG) and electroencephalogram (EEG) signals, a two-branch feature extraction structure based on an LSTM network is constructed. One branch is used to extract the temporal variation features shared by the two types of physiological electrical signals, while the other branch is used to extract the temporal features specific to each modality, including heart rate variability (HRV) index, spectral energy distribution features, functional connectivity features, etc., taking into account both the commonalities and characteristics of physiological signals.

[0076] For speech signals, a ResNet50 convolutional neural network is used to extract deep acoustic features from the spectrogram, obtaining deep acoustic features related to emotional expression, such as timbre, formants, and energy distribution contained in the spectrogram.

[0077] For facial expression videos, a network structure combining ResNet50 with squeeze-and-excitation (SE) channel attention mechanism and CPVT position encoding is used to extract facial micro-expression features and features of emotion-related areas such as the periorbital area and periorbital area. This effectively identifies abnormal manifestations unique to people with depression, such as stiff facial expressions, short-lasting smiles, and small facial movements.

[0078] It is worth noting that the aforementioned feature extraction networks can be replaced according to specific accuracy and efficiency requirements. For example, the speech feature extraction network can be replaced with a lightweight audio-oriented network (VGGish), and the facial feature extraction network can be replaced with a mobile-optimized lightweight face recognition network (MobileNet). In other words, the ResNet-50 network used for extracting speech and facial features can be replaced with lightweight networks such as MobileNet and EfficientNet to further reduce model complexity.

[0079] In this step, the five heterogeneous modal raw signals are uniformly transformed into dimension-aligned deep feature vectors, denoted as scale feature F1, ECG feature F2, EEG feature F3, speech feature F4, and facial expression feature F5, providing structured input for subsequent adaptive fusion.

[0080] Multimodal Feature Adaptive Fusion: This is the core invention of this work. It employs a dedicated fusion architecture comprised of multi-scale channel attention (MS-CAM) enhancement, modality relevance matrix construction and dynamic weight allocation, and attention feature fusion (AFF). This architecture intelligently integrates heterogeneous modal information across four progressive dimensions: single-modal feature enhancement, inter-modal correlation mining, dynamic weight adjustment, and multi-level feature deep fusion. The specific implementation process, limiting parameters, and calculation logic are as follows:

[0081] First, multi-scale channel attention (MS-CAM) enhancement is performed on each single-modal feature. The five single-modal features (denoted as F1-F5) extracted above are respectively input into independent multi-scale channel attention (MS-CAM) units for feature enhancement. Figure 3 This is a schematic diagram of the Multi-Scale Channel Attention (MS-CAM) unit structure of the present invention, demonstrating the parallel attention computation structure of global and local dual channels. As shown in the figure, the input feature X is processed in parallel by two paths. One path extracts global channel features through a bottleneck structure of global average pooling, pointwise convolution, ReLU activation, and pointwise convolution. The other path extracts local channel features through the same bottleneck structure (pointwise convolution, ReLU activation, and pointwise convolution). The two features are added element-wise and then activated by Sigmoid to generate channel attention weights. Finally, the weights are multiplied element-wise with the original input feature X to obtain the enhanced feature X'. In specific implementation, the unified input feature dimension is C×H×W, where the number of channels C can be configured as 128, 256, or 512 according to actual needs, and H=1 and W=1 to adapt to the feature formats of five heterogeneous modalities.

[0082] First, perform a global average pooling operation on the input features. The calculation formula is as follows:

[0083]

[0084] Where G(F) represents the global average pooling result of the input feature F, F represents the input unimodal feature, H represents the feature height, and W represents the feature width. This allows us to obtain the feature distribution patterns related to the depressive state in the global dimension.

[0085] Secondly, pointwise convolution (PWConv) is used to construct the bottleneck structure and aggregate local channel features. The convolution compression ratio is set to 1 / 16. First, pointwise convolution with a kernel size of Cr×C×1×1 is used to complete feature dimensionality reduction. After ReLU nonlinear activation, pointwise convolution with a kernel size of C×Cr×1×1 is used again to restore the channel dimension, preserving subtle local feature information. The formula for calculating local features is:

[0086]

[0087] Where L(F) represents local channel features, PWConv1 represents the first layer of pointwise convolution, PWConv2 represents the second layer of pointwise convolution, ReLU represents the linear rectified activation function, Cr represents the number of channels after dimensionality reduction, and Cr=C / 16.

[0088] Then, the global channel features and local channel features are added element-wise, and the results are normalized using the Sigmoid activation function to generate channel attention weights with values ​​ranging from 0 to 1. The calculation formula is as follows:

[0089]

[0090] Where W(F) represents the generated channel attention weights, L(F) represents the local channel features, and Sigmoid represents the sigmoid activation function.

[0091] Finally, the generated channel attention weights are multiplied element-wise with the original unimodal features to complete feature weighting enhancement, suppress noise and redundant features, and amplify sensitive features highly correlated with the depressive state, resulting in the enhanced unimodal features: .

[0092] Where F' represents the enhanced single-modal feature, W(F) represents the generated channel attention weight, and ⊗ represents the element-wise multiplication operation.

[0093] Second, construct the modal correlation matrix and complete the dynamic weight allocation. Based on the five-way enhanced single-modal features F'1 to F'5, construct the modal correlation matrix and solve for the dynamic weights. Use the Pearson correlation coefficient to calculate the correlation strength between different modal features pairwise, construct a 5×5 modal correlation matrix, and the matrix calculation expression is as follows:

[0094]

[0095] Where R(m, n) represents the Pearson correlation coefficient between the m-th mode and the n-th mode, corr represents the Pearson correlation coefficient calculation function, and F' m F' represents the enhancement feature of the m-th mode. n The enhancement feature represents the nth modality, where m and n are modality numbers, ranging from 1 to 5, corresponding to the scale, ECG, EEG, speech, and facial expression modalities, respectively. The correlation coefficient ranges from [-1, 1], and the larger the absolute value, the stronger the correlation between the two modalities.

[0096] The basic contribution weight of each mode is calculated based on the constructed modal correlation matrix. The weight calculation formula is as follows:

[0097]

[0098] Among them, W mR represents the basic contribution weight of the m-th mode, and |R(m,n)| represents the absolute value of the correlation coefficient between the m-th mode and the n-th mode.

[0099] To further adapt to the differences in various detection scenarios, scenario-adaptive correction rules are added: In resting-state detection scenarios, the base weights of the ECG and EEG modalities are multiplied by a factor of 1.5 to enhance the weight proportion of physiological electrical signals in the resting state; in task-oriented detection scenarios, the base weights of the speech and facial expression modalities are multiplied by a factor of 1.5 to amplify the role of behavioral and emotional features in the task execution process; the scale modal serves as the baseline weight and is not subject to factor correction. Finally, the corrected weights are normalized to ensure that the sum of the weights of the five modalities is 1, generating a dynamic weight vector adapted to the current detection sample and the current detection scenario.

[0100] Third, deep fusion of multimodal features is achieved by relying on the attention feature fusion (AFF) unit. Figure 4 This is a schematic diagram of the Attention Feature Fusion (AFF) unit structure of the present invention, illustrating the multi-level feature fusion logic based on dynamic weights. The first step involves performing element-wise summation on the five enhanced single-modal features to complete feature integration, obtaining the initial aggregated feature F. agg The calculation formula is:

[0101]

[0102] Among them, F agg F'1 represents the initial aggregated features, ⊎ represents the feature integration operation (element-by-element summation), and F'1-F'5 correspond to the enhanced features of the scale, ECG, EEG, speech, and facial expression modalities, respectively. The second step is to combine the initial aggregated features F... agg The data is fed into the MS-CAM module to generate a global fusion weight M. The value of weight M is strictly constrained to be between 0 and 1, and is used to balance the proportion of weighted single-modal features and initial aggregated features. The third step involves substituting the data into the multimodal fusion calculation formula specific to this invention to complete adaptive fusion.

[0103]

[0104] Extending to multimodal fusion as follows:

[0105]

[0106] Where X and Y represent different modal features, Z represents the final output fused feature vector, W1, W2, W3, W4, and W5 are the dynamic fusion weights for each modality; F'1, F'2, F'3, F'4, and F'5 are the enhanced single-modal features; F aggThe initial aggregated features are denoted as M; M is the global fusion weight. ⊗ represents element-wise multiplication, and ⊎ represents feature integration, i.e., element-wise summation.

[0107] Fourth, after completing the adaptive fusion, the output fusion feature vector Z is subjected to L2 normalization noise reduction to remove redundant noise data generated during the fusion process and retain the core features that are highly correlated with the depressive state in various modalities. At the same time, the final dynamic weight parameters of each modality are retained for subsequent source analysis of depression detection results, accurately labeling abnormal modalities that contribute significantly to the detection results, and realizing the interpretability of the detection results.

[0108] It is important to emphasize that the aforementioned complete multimodal feature adaptive fusion process is designed based on a modular fixed structure, defined parameter standards, and reproducible mathematical calculation steps. Any implementations that can be replicated by those skilled in the art based on the description of this embodiment without creative effort are within the scope of this invention. Furthermore, this fusion architecture features a lightweight design in terms of computational resources, with no redundant parameters, making it suitable for subsequent model compression and deployment requirements for mobile and wearable devices.

[0109] The depression classification and early detection process involves obtaining a fused feature vector Z through adaptive fusion of multimodal features. This vector is then fed into a Multi-Layer Perceptron (MLP) classifier to determine the depression state. The MLP consists of several stacked fully connected layers, and the final output layer, through an activation function, outputs the subject's depression probability and corresponding depression risk level.

[0110] In one embodiment, the module employs 5-fold cross-validation to optimize model parameters during training, using detection accuracy, sensitivity, and specificity as core evaluation metrics. The initial learning rate is set to 0.001, decaying by 10% every 5 training epochs. An early stopping strategy is used to prevent overfitting, with a patience value of 10. Based on the threshold classification criteria of the PHQ-9 scale, depression risk levels are divided into five levels: no depression, mild depression, moderate depression, moderate to severe depression, and severe depression. Simultaneously, based on the dynamic weight parameters of each modality retained during the fusion process, key abnormal modality prompts are output, clearly indicating that the abnormal source of the detection results is EEG, ECG, voice, or facial expressions, providing objective evidence for clinicians' diagnoses. Optionally, an intervention suggestion module can be added to the output end to generate personalized mental health guidance suggestions by combining depression risk level and abnormal modality characteristics, forming a complete closed loop from detection to evaluation to suggestion.

[0111] Figure 5This is a schematic diagram of the device framework of one embodiment of the present invention. As shown in the figure, the left area displays the visualization results of the raw data, including EEG sampling maps, ECG sampling maps, EEG frequency band complexity maps, and a pie chart of the LF / HF heart rate variability ratio; the middle area is a multimodal emotion index assessment module, which comprehensively calculates and displays the scale emotion index, ECG emotion index, EEG emotion index, voice emotion index, and facial expression assessment index; the bottom area outputs the final physical and mental health evaluation results and targeted intelligent health suggestions. This device can display the raw data and feature analysis results of each modality in real time, intuitively presenting the fusion process and detection basis of multimodal information, which is convenient for users to understand and for clinicians to refer to.

[0112] Optionally, based on the above classification module, a time-series prediction module can be added to provide early warning of the evolution of depression risk 3 to 6 months later based on the time-series change trend of multiple test data, providing a more forward-looking decision reference for clinical intervention.

[0113] To enable the device of this invention to achieve low-latency real-time detection on resource-constrained platforms such as wearable devices, mobile terminals, and portable campus screening hardware, in one embodiment, this invention also provides a lightweight deployment module:

[0114] Knowledge distillation technology is used to compress the trained large model by more than 60%. The large model is used as the teacher network, and the lightweight model is used as the student network. The knowledge of the teacher network is transferred through the distillation loss function. INT8 quantization is performed, and structured pruning technology is used to remove redundant convolutional kernels and fully connected layer neurons, further reducing the number of model parameters and computational complexity, and keeping the model's inference latency within 100ms. The lightweight model can be deployed on PCs, Android mobile devices, embedded terminals, and wearable hardware, supporting cross-scenario applications such as hospital clinical auxiliary diagnosis, large-scale campus screening, and home monitoring.

[0115] By employing the Z-score normalization preprocessing and domain adaptation techniques described above, data distribution offsets between different acquisition devices and detection scenarios can be effectively eliminated, ensuring that the accuracy of the model does not decrease by more than 5% when deployed across scenarios. If the model is only used for offline laboratory testing, this lightweight deployment step can be omitted.

[0116] Without departing from the core concept of this invention, there are also a number of alternative and extended schemes: the multi-scale channel attention (MS-CAM) unit can be replaced with other attention structures such as CBAM and ECA; the attention feature fusion (AFF) unit can be replaced with an adaptive gating fusion structure; and each single-modal feature extraction network can be replaced with different types of deep learning networks such as convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformers, according to actual accuracy and efficiency requirements.

[0117] Optionally, a transfer learning module can be integrated during the model training phase. The basic model can be pre-trained using a large-scale adult depression multimodal dataset to learn general depression feature representations. Then, fine-tuning can be performed on a limited-label adolescent-specific dataset to further improve the detection accuracy and generalization ability of the adolescent population in small sample scenarios.

[0118] Optionally, an intervention suggestion module can be added to the output of the depression status classification. The module takes the output depression risk level, key abnormal modalities and the contribution of each modality as input, and combines them with clinical treatment guidelines to generate graded and personalized mental health guidance suggestions, including daily emotion regulation methods, medical advice and follow-up suggestions, forming a complete closed loop of detection-assessment-suggestion.

[0119] According to the technical solution of this embodiment, those skilled in the art can stably achieve the following technical indicators: comprehensive accuracy of early depression detection ≥90%, sensitivity ≥88%, specificity ≥87%, cross-scenario deployment accuracy decay ≤5%, model inference latency ≤100ms, and detection accuracy of adolescents is 10%~15% higher than that of general adult models.

[0120] In summary, this invention provides a device for early detection of depression based on multimodal feature adaptive fusion. The device includes: a data acquisition module, a feature extraction module, an adaptive fusion module, a classification output module, and optional lightweight deployment and visualization interaction modules. Specifically, the data acquisition module simultaneously collects five heterogeneous modal data types: scales, ECG, EEG, voice, and facial expressions, and performs targeted noise reduction, correction, and Z-score normalization preprocessing on the collected raw data. The feature extraction module incorporates deep learning networks specific to each modality to extract uniform-dimensional deep feature vectors from the preprocessed data. The adaptive fusion module includes a multi-scale channel attention enhancement unit, a dynamic weight calculation unit, and an attention feature fusion unit, which sequentially perform multi-scale channel attention enhancement, dynamic weight allocation based on the modality correlation matrix, and attention feature fusion on the deep features to obtain a fused feature vector. The classification output module outputs a depression risk level and key abnormal modality indications based on the fused feature vector. The lightweight deployment module enables model compression and cross-terminal deployment. The visualization interaction module displays the raw data, feature analysis results, and the final detection report.

[0121] This device can operate independently on PCs, mobile terminals, and wearable devices, supporting multiple application scenarios such as clinical auxiliary diagnosis in hospitals, large-scale screening in schools, and home monitoring. It features high detection accuracy, fast response speed, interpretable results, and flexible deployment, providing reliable technical support for early screening and intervention of adolescent depression.

[0122] To better illustrate the present invention, numerous specific details have been provided in the detailed embodiments described above. Those skilled in the art should understand that the present invention can be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of the present invention.

[0123] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A device for early detection of depression based on adaptive fusion of multimodal features, characterized in that, include: The data acquisition module is used to acquire depression-related data from at least two different modalities of the participants; The feature extraction module is used to extract depth features for each modality; An adaptive fusion module is used to perform adaptive fusion on the depth features of each modality. The adaptive fusion module specifically includes: The multi-scale channel attention enhancement unit is used to perform multi-scale channel attention enhancement on the depth features of each modality to obtain the enhanced single-modality features; The dynamic weight calculation unit is used to construct a modality correlation matrix that reflects the pairwise correlation between each enhanced single-modal feature. Based on the sum of the correlation strength between each modality and all other modalities in the modality correlation matrix, the dynamic fusion weight of each modality is calculated. The attention feature fusion unit is used to perform weighted fusion of each enhanced single-modal feature according to the dynamic fusion weight, and perform attention feature fusion with the element-wise summation result of each enhanced single-modal feature to obtain a fused feature vector; The classification output module is used to output the depression risk level based on the fused feature vector.

2. The multimodal feature adaptive fusion early detection device for depression according to claim 1, characterized in that, The multi-scale channel attention enhancement unit is configured as follows: The input single-modal features are processed in two parallel paths: The first path first performs global average pooling, and then sequentially passes through a bottleneck structure consisting of the first pointwise convolution, the ReLU activation function, and the second pointwise convolution to extract global channel features; The second path directly extracts local channel features through a bottleneck structure composed of pointwise convolution, ReLU activation function, and pointwise convolution with the same structure. The global channel features and local channel features are added element by element, and then channel attention weights are generated by applying a Sigmoid activation function. The channel attention weights are multiplied element-wise with the input unimodal features to output the enhanced unimodal features.

3. The multimodal feature adaptive fusion early detection device for depression according to claim 1, characterized in that, The dynamic weight calculation unit is configured as follows: The correlation strength between any two enhanced single-modal features is calculated using the Pearson correlation coefficient, and a modal correlation matrix R is constructed. Each element R(m, n) of the modal correlation matrix R is the Pearson correlation coefficient between the m-th mode and the n-th mode, where m and n are the mode numbers. When there are 5 modes, the basic contribution weight W of each mode is calculated according to the following formula. m : Where |R(m, n)| represents the absolute value of the correlation coefficient R(m, n); and, The sum of the base contribution weights for all modalities is 1.

4. The multimodal feature adaptive fusion early detection device for depression according to claim 3, characterized in that, The dynamic weight calculation unit is also configured as follows: The basic contribution weights are adaptively adjusted based on the detection scenario as follows: When the detection scenario is a resting state, the weights of the ECG mode and / or EEG mode are multiplied by a factor of 1.

5. When the detection scenario is task-oriented, the weights of the speech modality and / or facial expression modality are multiplied by a factor of 1.

5. The base contribution weights of all modalities after correction are normalized so that the sum of the base contribution weights of all modalities is 1, thus obtaining the final dynamic fusion weights of each modality.

5. The multimodal feature adaptive fusion early depression detection device according to claim 1, characterized in that, The attention feature fusion unit is configured as follows: The initial aggregated feature F is obtained by summing the elements of each enhanced single-modal feature. agg ; The initial aggregation feature F agg Input a multi-scale channel attention unit to generate a global fusion weight M, wherein the value of the global fusion weight M ranges from 0 to 1; The enhanced single-modal features are summed element-wise according to the dynamic fusion weights of each modality to obtain the weighted aggregated features; The final fused feature vector Z is output using the following formula: in, This indicates element-wise summation; This represents element-wise multiplication; W1, W2, W3, W4, and W5 are the dynamic fusion weights for each modality; F'1, F'2, F'3, F'4, and F'5 are the enhanced single-modal features; F agg M represents the initial aggregation feature; M represents the global fusion weight.

6. The multimodal feature adaptive fusion early detection device for depression according to claim 1, characterized in that, The at least two different modes are selected from at least two of the following five categories: Scale data, electrocardiogram signals, electroencephalogram signals, voice signals, or facial expression videos.

7. The multimodal feature adaptive fusion early depression detection device according to claim 5, characterized in that, The extraction of depth features for each modality specifically includes: For the scale text data, a Transformer encoder is used to extract semantic feature vectors containing sentiment tendencies; For electrocardiogram (ECG) and electroencephalogram (EEG) signals, a dual-branch structure of long short-term memory (LSTM) network is used to extract the shared temporal variation features of the two types of physiological signals and the unique physiological features of each modality. For speech signals, a convolutional neural network is used to extract deep acoustic features from the spectrogram; For facial expression videos, a convolutional neural network with embedded channel attention mechanism and position encoding is used to extract micro-expression features of facial action units.

8. The multimodal feature adaptive fusion early depression detection device according to claim 1, characterized in that, The classification output module is also configured to: The dynamic fusion weights of each modality are output synchronously as abnormal modality source information of the detection results, and the key abnormal modalities with the highest contribution are marked.

9. The multimodal feature adaptive fusion early detection device for depression according to claim 1, characterized in that, Also includes: The lightweight deployment module is used to compress the trained model using at least one of the techniques of knowledge distillation, model quantization, and structured pruning. Furthermore, by employing Z-score normalization preprocessing and domain-adaptive technology, data distribution offsets between different acquisition devices and detection scenarios are eliminated.

10. The multimodal feature adaptive fusion early detection device for depression according to claim 1, characterized in that, Also includes: The visualization and interactive module is used to display the original data of each modality, feature analysis results, dynamic fusion weights of each modality, and the final depression risk level in real time.

Images