Adolescent depression auxiliary diagnosis model training method and system based on multi-modal data

Abstract

The application provides a kind of juvenile depression auxiliary diagnosis model training method and system based on multi-modal data, relating to mental health technical field.Firstly, the VR intelligent interaction scene is constructed, and the physiological data of juvenile depression and normal control subjects in the VR intelligent interaction scene are synchronously collected.The electroencephalogram data, heart rate variability data and eye movement data are preprocessed and feature extracted, and the single-mode depression recognition model is obtained through classifier training, and then the multi-modal depression recognition model and cross-modal depression recognition model are obtained.The juvenile depression auxiliary diagnosis model obtained by machine learning is helpful for early detection and timely treatment of juvenile depression.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of mental health, in particular relates to a kind of adolescent depression auxiliary diagnosis model training method and system. BACKGROUND

[0002] Depression refers to mental disorder caused by various reasons, and the main clinical features are mood depression, pessimistic and suicidal symptoms, etc.According to the data of the Health China Action (2019-2030) planning outline, the prevalence rate of depression in China is as high as 2.1%, and the age of onset is getting younger and younger. The global number of patients with depression has exceeded 300 million. It is estimated that by 2030, depression will become the world's first burden disease. In recent years, affected by the new crown epidemic, the prevalence rate of depression has risen sharply worldwide, and there has been a phenomenon of "new crown depression". The latest research data shows that during the past two years of epidemic, the incidence rate of adolescent depression has increased more than twice compared with that before the epidemic, and every 4 adolescents in the world have 1 with depressive symptoms, and the growth rate of adolescent prevalence has exceeded that of adults. In China, according to the China National Mental Health Development Report (2019-2020) released in March 2021, the detection rate of adolescent depression in 2020 was 24.6%, and the detection rate of severe depression was 7.4%. Adolescent depression has high incidence rate, high incidence of adverse events, slow onset and concealment, etc. However, the diagnosis and evaluation of adolescent depression at present mainly depend on symptomatology and scale evaluation, which often cannot effectively identify adolescent depression patients. The diagnosis of depression lacks a "gold standard" of biological diagnosis, and the diagnosis is largely uncertain, mainly relying on the subjective evaluation of psychiatrists. Statistics show that general practitioners can only correctly identify 47.3% of patients with depression, and there is a large proportion of missed diagnosis and misdiagnosis, and misdiagnosis leads to insufficient or excessive treatment. Therefore, it is urgent to develop an objective adolescent depression auxiliary diagnosis model based on multi-modal data, which is not dependent on the self-evaluation of the subject or the observation of others, as a "polygraph of depressive mood", to identify adolescent depression patients at the source. SUMMARY

[0003] Therefore, the technical problem to be solved by the present application is to provide an adolescent depression auxiliary diagnosis model training method and system based on multi-modal data, which can construct an effective adolescent depression recognition model, so as to objectively and accurately evaluate adolescent depression.

[0004] In a first aspect, the present application provides an adolescent depression auxiliary diagnosis model training method based on multi-modal data, comprising:

[0005] Step S1, constructing a VR intelligent interaction scene, and synchronously collecting physiological data of adolescent depression and normal control subjects in the VR intelligent interaction scene, the physiological data including electroencephalogram data, heart rate variability data and eye movement data;

[0006] Step S2, data preprocessing and feature extraction are performed on the electroencephalogram data, heart rate variability data and eye movement data to obtain electroencephalogram feature data, heart rate variability feature data and eye movement feature data;

[0007] Step S3, a single-modal depression recognition model is obtained by performing classifier training based on the electroencephalogram feature data, heart rate variability feature data and eye movement feature data respectively; the single-modal depression recognition model includes a depression recognition model based on the electroencephalogram data, a depression recognition model based on the heart rate variability data and a depression recognition model based on the eye movement data;

[0008] Step S4, a multi-modal depression recognition model is obtained by performing classifier training after feature layer fusion of the electroencephalogram feature data, heart rate variability feature data and eye movement feature data;

[0009] Step S5, the electroencephalogram feature data, heart rate variability feature data and eye movement feature data are respectively taken as inputs of an autoencoder to generate respective shared representations; only one kind of feature data is provided during classifier training, and the other two kinds of feature data are tested on the trained classifier to obtain a cross-modal depression recognition model.

[0010] Further, in the step S2,

[0011] The electroencephalogram data preprocessing includes: performing 0.5-100Hz band-pass filtering on the original electroencephalogram signal to remove low-frequency drift and high-frequency noise; applying a 50Hz notch filter to eliminate power supply interference; performing signal denoising through discrete wavelet transform; calculating SNR and RMSE to evaluate signal quality; and finally separating the processed signal into five frequency bands of Delta, Theta, Alpha, Beta and Gamma for power spectral density analysis;

[0012] The heart rate variability data preprocessing includes: applying 0.04-5Hz band-pass filtering to the original photoplethysmogram signal to remove baseline drift and high-frequency noise; performing signal denoising through discrete wavelet transform; identifying heartbeat intervals through a peak detection algorithm; and extracting time domain and frequency domain indicators;

[0013] The eye movement data preprocessing includes: if the missing value of a data tuple in the eye movement data exceeds a set threshold, the data tuple is discarded; and if the missing value of a data tuple does not exceed the set threshold, the missing value is filled with an average value or a median value.

[0014] Further, in the step S2, the feature extraction adopts a correlation-based feature selection method combined with a best-first search strategy.

[0015] Further, before the step S3, a Gaussian filter formula is used for denoising, Gaussian noise is added for data enhancement, a weighted cross-entropy loss function is used for sample rebalancing, and a support vector distance discriminant function formula is used for edge sample identification and resampling; the significance P value of the evaluation feature is defined, the features with insignificant differences are removed through the difference test between the evaluation feature and the target label, and finally K-fold cross-validation is performed to determine the optimal feature subset based on the comprehensive performance on the training set.

[0016] Further, in the step S3, any feature data in the electroencephalogram feature data, heart rate variability feature data and eye movement feature data is taken as the input layer of the autoencoder, noise is added in the original input, shared representation is generated through encoding, and the original input is reconstructed through decoding.

[0017] Further, in the step S4, the feature layer fusion step is: connecting the electroencephalogram feature data, heart rate variability feature data and eye movement feature data directly together, inputting into the autoencoder to generate shared representation; using an unsupervised back propagation algorithm to fine-tune the weights and biases of the autoencoder to generate the final shared representation for actual training of the classifier.

[0018] Further, the fused multi-modal feature vector is denoted as:

[0019] z fusion ∈R d

[0020] wherein z fusion is the feature vector after multi-modal fusion;

[0021] Four parallel sub-models f1, f2, f3 and f4 are defined, and the output prediction probability is:

[0022] pi=fi(·),i=1,2,3,4

[0023] wherein f i is the i-th sub-model for outputting depression prediction at different angles;

[0024] p i is the prediction probability output by the i-th sub-model;

[0025] The fusion is the depression prediction probability:

[0026]

[0027] wherein,

[0028] where p(y = 1 | z fusion ): represents the probability of the sample belonging to class 1, class 1 being depression, under the condition that z fusion is given;

[0029] λ i is the weight coefficient of the i-th sub-model, satisfying

[0030]

[0031] p i is the output probability of the i-th sub-model, specifically:

[0032] p1 is the output of the standard MLP sub-model;

[0033] p2 is the output of the modal attention mechanism model;

[0034] p3 is the output of the contrast enhancement model;

[0035] p4 is the output of the auxiliary task model;

[0036] The sub-model f1 is a standard MLP model, which is used to model the fusion features non-linearly:

[0037] p1 = f1(z fusion ) = Softmax(W1 · z fusion + b1)

[0038] W1 is a weight matrix;

[0039] b1 is a bias vector;

[0040] Softmax(·) is used to transform the output into a probability;

[0041] The sub-model f2 is a modal attention modeling MLP, which is used to adjust the modal weight:

[0042]

[0043] p2 = f2(z fusion ) = Softmax(W2 · z fusion + b2)

[0044] where:

[0045] z i is the representation of the i-th modal;

[0046] w is a trainable vector parameter in the attention mechanism;

[0047] w is a trainable weight vector used to calculate the importance of different modalities;

[0048] tanh(·) is the hyperbolic tangent function, which introduces a nonlinear transformation as an activation function for input features;

[0049] exp(·) is the exponential function, which is used to amplify the difference and form a softmax format;

[0050] α i represents the attention weight of the i-th modality, the larger the value, the higher the importance of the modality to the final classification task;

[0051] M is the total number of modalities;

[0052] The sub-model f3 is a contrast enhancement model, which uses the difference between the positive and negative stimulus responses in the emotional task as an enhanced feature modeling:

[0053]

[0054] z f (positive) is the fusion feature representation extracted by the model under positive emotional stimulus;

[0055] z f (negative) is the fusion feature representation extracted by the model under negative emotional stimulus;

[0056] Δz is the difference vector of the two states, which is used to capture the response intensity of the subject to different emotional stimuli, and is the enhanced feature;

[0057] [z fusion ,Δz] represents the concatenation of the original fusion feature and the emotional difference feature, which is used to enhance the input expression ability of the model;

[0058] w3 is the weight matrix of the sub-model f3;

[0059] b3 is the bias term;

[0060] f3(·) is the contrast enhancement modeling module;

[0061] p3 is the output probability of the sub-model f3;

[0062] The sub-model f4 is an auxiliary task modeling module, which is used to predict the individual's behavior variables in parallel while performing the main task:

[0063] f4=f4 (main) (z fusion )=Softmax(W4·z fusion +b4)

[0064] where W4 and b4 are learnable parameters;

[0065] The output is a probability p4, representing the predicted probability that the sample is depressed;

[0066]

[0067] r^ is the predicted reaction time;

[0068] s^ is the predicted behavior score;

[0069] w r ,w s and b r ,b s are trainable parameters of the auxiliary task;

[0070]

[0071] The mean squared error MSE is calculated for each of the two behavior variables;

[0072] λ r ,λ s represent the weighting coefficients of the auxiliary loss;

[0073] The outputs of the four sub-models are combined by weighting:

[0074]

[0075] Beta distribution is added to model uncertainty:

[0076] α = exp(W α ·z fusion ), β = exp(W β ·z fusion )

[0077]

[0078] Beta distribution parameters α and β are used to describe the confidence of the model in the prediction result;

[0079] is the expectation of the prediction result;

[0080] Var(y) is the variance of the prediction result, used to measure uncertainty, and the larger the value represents the more uncertain the model.

[0081] Further, course learning and adversarial training mechanism are used to optimize the model.

[0082] Further, the VR intelligent interaction scene uses A-Frame framework to build Web VR environment, and builds a mind dialogue space through Claude API.

[0083] In another aspect, the present application also provides a multi-modal data-based adolescent depression auxiliary diagnosis model training system, and the technical scheme of the system is as follows:

[0084] A virtual reality subsystem is configured to construct a VR intelligent interaction scene, and to synchronously collect physiological data of adolescent depression and normal control subjects in the VR intelligent interaction scene, wherein the physiological data includes electroencephalogram data, heart rate variability data, and eye movement data.

[0085] A data processing subsystem is configured to perform data preprocessing and feature extraction on the electroencephalogram data, heart rate variability data, and eye movement data, to obtain electroencephalogram feature data, heart rate variability feature data, and eye movement feature data.

[0086] A single-modal depression recognition model training subsystem is configured to perform classifier training based on the electroencephalogram feature data, heart rate variability feature data, and eye movement feature data, respectively, to obtain a single-modal depression recognition model, wherein the single-modal depression recognition model includes an electroencephalogram data-based depression recognition model, a heart rate variability data-based depression recognition model, and an eye movement data-based depression recognition model.

[0087] A multi-modal depression recognition model training subsystem is configured to perform classifier training after feature layer fusion of the electroencephalogram feature data, heart rate variability feature data, and eye movement feature data, to obtain a multi-modal depression recognition model.

[0088] A cross-modal depression recognition model training subsystem is configured to take the electroencephalogram feature data, heart rate variability feature data, and eye movement feature data as inputs of automatic encoders respectively, to generate respective shared representations, and to provide only one kind of feature data during classifier training, and to test the other two kinds of feature data on the trained classifier, to obtain a cross-modal depression recognition model.

[0089] Further, the virtual reality subsystem includes a Biopac MP160 physiological multichannel instrument and an aSee A8 portable remote eye tracker, which are configured to obtain the electroencephalogram data, eye movement data, and heart rate variability data.

[0090] Beneficial effects:

[0091] The multi-modal data-based adolescent depression auxiliary diagnosis model training method provided by the present application constructs a more ecological and enhanced irrelevant variable control scene through virtual reality technology, accurately collects electroencephalogram, heart rate variability, and eye movement data of adolescent depression and normal control subjects, and builds a more effective adolescent depression recognition model.

[0092] The application is based on a single-mode depression recognition model of electroencephalogram, eye movement and heart rate variability characteristics, and the single-mode electroencephalogram / eye movement / heart rate variability characteristics are input into a denoising autoencoder to generate respective shared representations for training a classifier to obtain a single-mode depression recognition result.

[0093] By utilizing the complementarity of different modalities, a multi-modal depression model is trained based on electroencephalogram, eye movement and heart rate variability characteristics, and two modal fusion strategies (feature fusion and hidden layer fusion) are used to fuse the feature layers of electroencephalogram, eye movement and heart rate variability data, thereby improving the classification accuracy and constructing a more accurate depression recognition model.

[0094] The cross-modal adolescent depression model training of the application obtains whether it has a strong correlation with adolescent depression and a weak correlation with the signal form of electroencephalogram / eye movement or heart rate variability through the shared representation learned by the autoencoder.

[0095] In addition, the adolescent depression auxiliary diagnosis model training system based on multi-modal data provided by the application has the same technical effects as the above training method. BRIEF DESCRIPTION OF DRAWINGS

[0096] In order to make the content of the application more easily understood, the application will be further described in detail below according to specific embodiments of the application and in conjunction with the accompanying drawings.

[0097] Figure 1 The flowchart of the adolescent depression auxiliary diagnosis model training method of embodiment 1 of the application is shown in the figure.

[0098] Figure 2 The support vector machine (SVM) model flowchart used in the adolescent depression auxiliary diagnosis model training method of embodiment 1 of the application is shown in the figure.

[0099] Figure 3 The normality test calculation result graph of the collected physiological indicators of embodiment 1 of the application is shown in the figure.

[0100] Figure 4 The ROC curve graph of embodiment 1 of the application is shown in the figure. DETAILED DESCRIPTION

[0101] The application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments. The principles and characteristics of the application are described below in conjunction with the accompanying drawings, and it should be noted that the embodiments in the application and the characteristics in the embodiments can be combined with each other without conflict. The embodiments are only used to explain the application and are not used to limit the scope of the application.

[0102] With the development of wearable technology, real-time, non-invasive physiological signal acquisition and analysis has become possible, which has prompted more and more research to focus on the feasibility of physiological signals from the body itself for monitoring and assessing stress, in order to achieve objective measurement of psychological stress. Existing research explores the specific manifestations of depression through physiological signals and external behavioral signals, and seeks objective indicators for the assessment of depression. Domestic and foreign research shows that non-invasive, objective physiological signals such as electroencephalogram (EEG), heart rate variability, and eye movement signals can reflect a person's psychological state to varying degrees. EEG is an objective and reliable method for assessing brain function, with high sensitivity, low cost, and portability. Some researchers have explored the application of EEG in the identification of depression. Previous studies have shown that the power spectrum of the three wave bands (alpha, beta, and theta) and the full wave band of EEG can achieve a classification accuracy of 71.7% and 88.6%, respectively, using support vector machines, achieving good results in identifying depression. Numerous studies have confirmed that the use of EEG technology to examine the activity of alpha waves (8-13 Hz) in the left and right frontal lobes can analyze the asymmetry of resting frontal EEG activity, which can be used as an important neural indicator for measuring depression and has good predictive effects. However, the explanation of frontal lobe lateralization is still at a relatively rough stage, and almost no research has involved and accurately analyzed the specific meaning of the lateralization score. Therefore, the use of frontal EEG lateralization score to distinguish between depression and non-depression groups is still difficult to implement in clinical practice. Recent research has focused on phase-amplitude coupling (PAC), which involves some cognitive processes and mental health. Studies have found that cross-coupling between different frequency neural oscillations reflects the synchronization between local and global networks in the brain, and abnormal patterns of PAC are associated with depression in adults. Decreased theta-gamma PAC in the resting state may be a biomarker of poor mental health, particularly depression, and sustained changes in PAC may be the underlying mechanism of TMS treatment, with a decrease in depressive symptoms after TMS associated with an increase in PAC. Therefore, further research is needed on frontal lobe lateralization and PAC, and the establishment of a more accurate depression prediction model based on EEG is an urgent problem to be solved.

[0103] Heart rate variability (HRV) is a commonly used non-invasive biomarker, which is easy to measure and can be collected non-invasively with wearable devices. HRV can not only reflect the influence of the internal and external environment on the cardiovascular system, but also reflect the corresponding adjustment of the cardiovascular system under the regulation of the autonomic nervous system and the humoral regulation. Because the change of emotional state directly affects the autonomic nervous system and the humoral regulation, the emotional change of an individual can be reflected in HRV, that is, HRV can be used for emotion recognition. HRV refers to the small difference between successive heart periods, which is jointly regulated by sympathetic and parasympathetic nerves, and reflects the balance of autonomic nervous function. When sympathetic nerve activity decreases or vagus nerve activity increases, HRV increases, and vice versa. The neural and humoral factors involved in HRV may be a valuable indicator for predicting sudden cardiac death and arrhythmic events: decreased parasympathetic activity, increased sympathetic activity, decreased HRV level, and decreased ventricular fibrillation threshold, which is prone to malignant arrhythmia. Existing studies generally find that depression is closely related to the decrease of HRV, and the HRV of patients with severe depression is lower than that of patients with mild depression, while the successful treatment of depression will be accompanied by the increase of HRV. However, some studies have found that the decrease of HRV in patients with depression may be related to other heart diseases, not caused by single depression, and the HRV index of patients with depression has significant fluctuation changes with age. A large number of studies have found that the abnormality of HRV is related to the bidirectional correlation between depression and cardiovascular disease, and patients with depression are more prone to cardiovascular disease than healthy individuals, and vice versa. Therefore, there is no uniform and effective standard for predicting depression by using HRV index, and whether the HRV index can be reliably associated with specific life phenomena in other fields is still unknown, and the present application further discusses the reliability and sensitivity of HRV as a physiological indicator for depression recognition. Eye movement (EM) is considered to be another effective physiological signal in depression recognition, and a large number of studies on eye tracking technology have shown that the characteristics of affective disorders are attentional bias to emotional stimuli. Eye tracking technology allows relatively direct and continuous measurement of visual attention, and is a non-invasive and economical and simple method with high temporal and spatial resolution. The eye movement abnormalities of patients with depression have been widely studied, and compared with the healthy control group, the patients with depression have abnormal horizontal pursuit eye movement, weak correlation between pursuit and saccade movement, high blink rate, and abnormal saccades, indicating that the eye movement system of the patients is impaired. Patients with depression show atypical visual scanning patterns in free viewing tests, characterized by longer fixation time, more fixation times, fewer saccades, and attentional bias to negative emotional faces. However, studies based on eye movement tests such as pursuit tests show inconsistent findings. Therefore, many specific features of eye movement of patients with depression need to be further explored and in-depth studied.

[0104] In recent years, virtual reality (VR) has shown great potential in the assessment and diagnosis of mental illness. Virtual environments can show significant differences in outcome measures between patients with mental illness and healthy controls. Virtual reality technology provides a multi-sensory three-dimensional environment that fully immerses users in a simulated world. Users perceive three-dimensional stereoscopic images through a head-mounted display and determine spatial positions in the visual environment through motion tracking sensors in the headset. At the same time, users can hear sounds through headphones and interact with virtual objects using input devices such as joysticks, wands, and data gloves. Compared with passive media such as radio and television, VR can provide higher levels of cognitive, social, and physical interaction, thereby enhancing the impact of the virtual environment on users. Virtual reality environments can stimulate and simultaneously measure symptoms of mental illness, helping to improve the reliability of mental illness assessments. Adding other measurement methods to VR measurements, such as physical activity, eye movement, distance tracking, and physiological measurements, can further improve the objectivity of VR measurements. These measures can usually be obtained without distracting the participant's attention, so immersion in the VR environment is not disturbed. In addition, the introduction of VR has the advantages of creating a standardized environment to ensure consistency and replacing standard laboratory environments with immersive technology to motivate participants. Currently, VR technology has been shown to compensate for the shortcomings of subjective assessments and neurophysiological measurements in psychological assessments, and many studies have found significant correlations between virtual reality outcome measures for patients with schizophrenia, hyperactivity disorder, and obsessive-compulsive disorder and traditional diagnostic measures. However, there are still few studies on adolescent depression, and the symptoms of depressed patients in virtual environments are not clear, and predictive models for depression based on VR technology are still to be developed.

[0105] In summary, the present application aims to develop an objective, accurate, convenient and practical method for assessing adolescent depression. Based on the EEG, heart rate, and eye movement variability characteristics of depressed and normal control subjects, combined with VR measurement technology, a more effective depression recognition model is expected to be constructed.

[0106] The present application has innovation and breakthrough in the research of non-invasive collection and analysis of physiological signals of depression. The main method is to construct a more ecological and enhanced irrelevant variable control scene through virtual reality technology (VR), accurately collect EEG, heart rate variability, and eye movement data of depressed and normal control subjects, and construct a more effective depression recognition model.

[0107] Finding non-invasive EEG biomarkers of depression is important as it can help diagnose the disease in a more objective way, as it is often diagnosed by using questionnaires that rely on the professional and the patient's subjectivity. But it is also difficult because depression presents different symptoms, has a high incidence, especially with anxiety, resulting in inconsistent biomarkers found in the literature. Given the complexity of depression, it is important to understand the latest findings of its biomarkers, and multi-modal / cross-modal depression recognition based on EEG, heart rate variability features, and eye movements can provide key clues.

[0108] In addition, in existing research, most of the experiments on multi-modal / cross-modal depression recognition occur in the laboratory, which effectively controls irrelevant variables but greatly weakens the ecological type required by the experiment. The present invention uses virtual reality technology (VR) to construct a more ecological experimental environment while controlling irrelevant variables, making the research results more practical and effectively applied to reality.

[0109] Embodiment 1

[0110] The present embodiment is a method for training a model for assisting in the diagnosis of adolescent depression based on multi-modal data, comprising the following steps:

[0111] 1. The key technology solved by this method is VR technology: combined with human-computer interaction, user experience, combined with generative artificial intelligence technology, activate the visual, auditory and other multi-sensory channels of adolescents, promote adolescents to immerse in a friendly virtual environment, and then create a more ecological and enhanced irrelevant variable control scene to achieve the purpose of accurately collecting EEG, heart rate variability and eye movement data of the subjects.

[0112] A-Frame framework is used to build a Web VR environment, and Claude API is used to realize functions such as "expert's voice" to build a mental dialogue space and help identify students' depression. This scene integrates the immersive experience of virtual reality technology and the natural dialogue ability of large language models, creating a safe and private virtual space through natural environment, soothing sound effects and interactive elements, promoting students to express their recent pain and hope, and providing important information for identifying depression.

[0113] While the subjects are immersed in the VR scene, the Biopac MP160 physiological multichannel instrument (EEG and PPG acquisition system), aSee A8 portable remote eye tracker will be connected with the subjects at the same time; portable electroencephalogram, pulse and other physiological data are collected to monitor the psychological process and dynamic changes, track the activation and relaxation state of the brain region of adolescents, understand the attention preference of adolescents to the VR intelligent interaction scene, and study the changes of the autonomic nervous system of adolescents. This embodiment uses electroencephalogram, eye movement and heart rate variation synchronous acquisition network to ensure that the electroencephalogram and heart rate variation data of the subjects during free browsing of emotional faces are recorded simultaneously with millisecond precision synchronization, which is the basis for meaningful synchronization analysis of electroencephalogram, eye movement and heart rate variation.

[0114] As a preferred embodiment, the physiological data collected in the VR intelligent interaction scene is corrected. Although VR has great potential in the assessment and diagnosis of mental illness, it can provide a multi-sensory three-dimensional environment that completely immerses people in a simulated world, but we found in our research that there are still some differences between adolescent depression and normal control subjects in the VR intelligent interaction scene and the real environment. When using the VR intelligent interaction scene, adolescents will have a delay in entering the virtual environment at the beginning of the experiment, and will also have a situation of being free from the original environment in the experiment. Thus, the psychological response is different from the real environment, affecting the accuracy of the collected experimental data.

[0115] This application fully considers this influence, and corrects the physiological data by adding a VR influence factor to reduce the psychological influence difference between depression and normal control subjects in the real environment and the VR intelligent interaction scene.

[0116] The VR influence factor is obtained by the following steps:

[0117] 1) Construct a typical real environment for comparison, and establish a typical VR intelligent interaction scene identical to the typical real environment;

[0118] 2) Conduct experiments on depression and normal control subjects in the typical real environment and the typical VR intelligent interaction scene respectively, and collect corresponding physiological data;

[0119] 3) Analyze and compare the physiological data of depression and normal control subjects in the typical real environment and the typical VR intelligent interaction scene, and obtain the VR influence factor.

[0120] For example, an easy-to-implement comparative test of a typical real environment about heart rate variability data is constructed, part of the depressive and normal control subjects are selected, and heart rate variability data is collected. The collected heart rate variability data is analyzed, if the two are linear, the simple ratio of heart rate variability data in the two cases can be used as the VR influence factor. In the formal test, the VR influence factor is multiplied by the heart rate variability data to obtain heart rate variability data closer to the real environment.

[0121] If the two are nonlinear, the data relationship or correspondence table can be established by data fitting, which is used as the VR influence factor. In the formal test, the heart rate variability data is added to the VR influence factor for correction.

[0122] 2. After data collection, pre-processing operation is performed, specifically including:

[0123] 1) Electroencephalogram preprocessing

[0124] Because electroencephalogram is a relatively weak, low frequency and weak anti-interference electrical physiological signal, it may be disturbed by physiological electrical signals such as electrooculogram and electromyogram during collection, affecting the signal quality. Therefore, before analyzing and processing the data, it is necessary to remove the noise in the original electroencephalogram.

[0125] The electroencephalogram preprocessing adopts a multi-step optimization process, including linear interpolation processing NaN value, applying 0.5-100Hz band-pass filter to remove baseline drift and high-frequency electromyographic noise, and performing 50Hz notch filter to eliminate power supply interference when the sampling rate is sufficient. Artifact detection and repair are performed on the filtered signal to improve detection accuracy. Discrete wavelet transform (DWT) algorithm is used for deep denoising, which is particularly effective in removing complex artifacts such as electrooculogram. The processed signal is decomposed into five standard frequency bands: Delta (0.5-4Hz), Theta (4-8Hz), Alpha (8-13Hz), Beta (13-30Hz) and Gamma (30-50Hz), and the power of each frequency band is calculated through optimized power spectral density analysis. The pre-processing quality is evaluated by signal-to-noise ratio and root mean square error to ensure the reliability of subsequent feature extraction and analysis.

[0126] 2) Heart rate variability preprocessing

[0127] The photoplethysmography (PPG) signal needs to go through a series of processing steps to obtain accurate results when used to calculate heart rate variability. The PPG signal is usually in the frequency range of 0.5-10 Hz, and various noises, including motion artifacts, ambient light interference, and baseline drift, are mixed in during the acquisition process, which can be removed by wavelet transform, band-pass filtering, median filtering, and other methods. The processed signal needs to be peak detected to obtain the pulse interval sequence, and then various heart rate variability indicators can be calculated, including time domain, frequency domain, and other analysis methods.

[0128] 3) Eye movement signal preprocessing

[0129] This embodiment uses three classic preprocessing methods in data mining: filling missing values, removing outliers, and data standardization. During the experiment, if the subject does not focus on the scene or blinks too frequently or closes his eyes for too long, the eye tracking device may lose the capture of the subject's pupil, resulting in missing values in some data tuples recorded. Under normal circumstances, there are two different strategies to deal with these data with missing values. If a data tuple contains a large number of missing values (such as more than 30% of the attributes), the tuple will be discarded. If a data tuple contains a small number of missing values (such as less than 30% of the attributes), the average or median of the subject's attribute is used to fill in the missing values.

[0130] In this embodiment, the electroencephalogram signal, eye movement signal and electrocardiogram signal are collected by Biopac MP160 physiological multichannel instrument and aSee A8 portable remote eye tracker, and then data preprocessing and feature extraction are performed respectively.

[0131] The extracted electroencephalogram and eye movement features are selected by the Bestfirst feature selection method based on CFS, and the selected features are used for subsequent single / multi / cross-modal depression recognition research.

[0132] The experimental process is as follows:

[0133] 51 adolescent patients with depression and 64 normal control adolescents (all subjects are right-handed, with normal vision / corrected vision, no astigmatism, no any brain or cardiovascular disease, and the subjects in the depression group have not been diagnosed with bipolar disorder, schizophrenia, autism spectrum disorder, attention deficit and hyperactivity disorder or other serious mental disorders at present or in their lifetime, and the subjects in the normal group have not been diagnosed with mental illness at present or in their lifetime) are recruited, and their electroencephalogram, eye movement and heart rate variability data are collected in a VR environment. Through data analysis, specific differences of depression on the three markers are obtained, and then modeling is performed.

[0134] After informing the subjects of their rights and filling out the informed consent form, the following are performed:

[0135] (1) Baseline test;

[0136] (2) VR scenario experience: By immersing the subjects in the VR scenario and completing the dialogue task, the state of the participants is monitored in real time, while monitoring the EEG, eye movement and heart rate variability data.

[0137] VR is a web-based VR interactive scenario with a beautiful environment,

[0138] The subjects enter a VR immersive environment under the guidance of the responsible person, in which the subjects can freely explore the VR space; dialogue with the elves in the VR world, and express the feelings, thoughts and confusion in the hearts of the subjects; interact with the scene, and activities away from pain, etc.

[0139] (3) Post-test: After the depression and control subjects complete the experiment, the EEG, eye movement, and heart rate variability data collected will be preprocessed and feature extracted for subsequent modeling.

[0140] 3. Single-mode depression recognition

[0141] In the single-mode depression recognition research based on EEG, EM, and HRV features, the single-mode EEG / EM / HRV features are used as the input layer of the denoising autoencoder, noise is added to the original input, and the shared representation (hidden layer) of EEG / EM / HRV is generated through the encoding process, and the original input is reconstructed through the decoding process. The shared representation generated is the high-level feature learned by the autoencoder related to depression, which is used as the input data in the classifier training process.

[0142] For example, in the training process of the classifier, the embodiment adopts a verification strategy of 5-fold cross-validation repeated 3 times. Specifically, the data of 115 subjects (64 control group and 51 depression group) are randomly divided into 5 non-overlapping subsets, and each time 4 subsets (92 data) are selected as the training set to train the SVM classifier, and the remaining 1 subset (23 data) is used as the test set for evaluation, and the classification accuracy is obtained. The process is repeated 3 times to ensure the stability of the results, and the average classification accuracy of the 3 experiments is taken as the evaluation index.

[0143] To comprehensively evaluate the performance of the model, we use the trained SVM classifier to perform final testing on all 115 data, and calculate the overall classification accuracy. At the same time, the classification accuracy of the control group and the depression group is recorded to evaluate the performance difference of the model in different categories. All experimental processes maintain strict sample balance between groups to ensure the reliability of model evaluation.

[0144] 4. Multi-modal depression recognition

[0145] The present embodiment uses a multimodal denoising autoencoder (MDAE) to fuse feature layers of electroencephalogram, eye movement and heart rate feature data. Specifically, the present application will use two modal fusion strategies (feature fusion and hidden layer fusion).

[0146] a) Structure diagram of feature fusion strategy: connect EEG features, EM features and HRV features together directly, input into the autoencoder to generate shared representation, then use unsupervised backpropagation algorithm to fine-tune the weights and biases of the autoencoder, and finally the generated shared representation is used for actual training of the classifier.

[0147] b) Structure diagram of hidden layer fusion strategy: input EEG features, EM features and HRV features into the autoencoder respectively to generate respective shared representations, still use unsupervised backpropagation algorithm to fine-tune the weights and biases of the autoencoder, connect the shared representation of EEG and the shared representation of EM directly to synthesize a new shared representation, and finally this synthesized shared representation is used as input data of the six classifiers for training of the classifiers.

[0148] The training process of the classifier is the same as the method in single-mode depression recognition, and a verification strategy of 5-fold cross-validation and 3 times repetition is adopted. The entire subject data is randomly divided into 5 folds, 4 folds are used to train the SVM classifier each time, and the remaining 1 fold is used for testing, and the process is repeated 3 times to ensure stability. Finally, the trained model is used to evaluate on the entire data set, and the average classification accuracy is calculated as the final result of multimodal depression recognition.

[0149] 5. Cross-modal depression recognition

[0150] Respectively use electroencephalogram, eye movement and heart rate variability features to learn high-level features (shared representation), and use EEG (delta, theta, alpha, beta, gamma band and full band) features and EM features as inputs of the autoencoder to generate respective shared representations. In the training phase of the classifier, only single modal data is provided, and other modal data is tested on the trained classifier.

[0151] In the present embodiment, it is mainly divided into three parts:

[0152] EM training: use the shared representation generated by EM as training data, and use the shared representation generated by EEG / HRV as test data.

[0153] EEG training: use the shared representation generated by EEG as training data, and use the shared representation generated by EM / HRV as test data.

[0154] HRV training: the shared representation generated by HRV as training data, and the shared representation generated by EEG / EM as test data.

[0155] The training process of the classifier is the same as that in single-mode and multi-mode depression recognition, and the strategy of 5-fold cross-validation and 3 times of repetition is adopted. Similarly, in order to verify the stability of the model, the SVM training and evaluation process is repeated 3 times, and the average accuracy and standard deviation of the 3 classification accuracies are calculated as the final classification result of multi-mode depression recognition.

[0156] The classifier algorithm of the embodiment adopts a support vector machine (SVM) model, and the specific flowchart is shown in Figure 2 .

[0157] The 55 physiological indicators collected were subjected to normality test, and the significance level p < 0.05 was marked. The indicators conforming to the normal distribution were subjected to independent sample t test, and the indicators not conforming to the normal distribution were subjected to Mann-Whitney U test. The p value less than 0.05 indicates that there is a significant difference in the indicators between the two groups of people.

[0158] Specific formula:

[0159] t value formula:

[0160]

[0161] Wherein:

[0162] M1 and M2 are the means of the two samples.

[0163] o is the pooled variance (combined variance):

[0164]

[0165] o and are the variances of the two samples, and n1 and m2 are the sample sizes.

[0166] o degrees of freedom df = n1 + n2 - 2.

[0167] The calculation results are shown in Figure 3 .

[0168] Note: (*p < 0.05, **p < 0.01, ***p < 0.001,

[0169] A binary classification model capable of effectively distinguishing between "normal" and "depressed" individuals was constructed using Support Vector Machine (SVM). The model's hyperparameters were fine-tuned through 5-fold cross-validation, ultimately selecting the Radial Basis Function (RBF kernel) with parameters C = 100 and gamma = 0.1. Physiological indicators showing significant differences (p < 0.01) at the 0.01 level were included in the model construction. Overall, with a class weight of 1, the model achieved an accuracy of 81.74% and an AUC (Area Under the ROC curve) of 0.921, demonstrating good classification ability. (See...) Figure 4 .

[0170] To obtain the optimal feature result:

[0171] avg_saccade_distance(px), avg_saccade_duration(s), theta_beta_ratio, lfhf ratio

[0172] In a preferred embodiment, the present invention employs the following specific steps for model training:

[0173] (1) Data processing

[0174] Beyond conventional denoising and standardization, this method combines data augmentation and sample rebalancing techniques to effectively improve model performance under conditions of limited data or class imbalance. Data augmentation expands the training sample space by adding noise and slicing, simulating more diverse physiological states and improving the model's generalization ability. Sample rebalancing adjusts class weights in the loss function, making the model pay more attention to minority class samples during training, mitigating the bias problem caused by class imbalance. Furthermore, an edge sample identification and resampling strategy is introduced, utilizing the support vector information of SVM to identify and distinguish samples with blurred boundaries, and increasing the proportion of such samples in the training set through local synthesis and interpolation.

[0175] In terms of feature selection, a multi-level statistical method was used for feature screening. By combining the p-value to evaluate the significance of features and the ANOVA F test to evaluate the correlation between features and target labels, redundant features with insufficient information were eliminated, and features with significant discriminative ability were selected. Finally, K-fold cross-validation was used to determine the optimal feature subset based on the overall performance on the training set, in order to reduce the interference of noisy features and improve the stability and interpretability of the model.

[0176] (1.1) Noise Reduction

[0177] Denoising employs a Gaussian filter formula to clean up invalid noise in the real data, improving data quality and allowing the model to learn meaningful features more accurately.

[0178]

[0179] (1.2) Data Augmentation

[0180] Data Augmentation adopts Gaussian noise addition formula to artificially increase disturbance, so that the model can learn to correctly classify under slight changes, thereby improving robustness and generalization ability:

[0181]

[0182] (1.3) Sample Rebalancing

[0183] Sample Rebalancing adopts weighted cross-entropy loss function to adjust the model's attention to different categories, alleviating the bias problem caused by class imbalance:

[0184]

[0185] (1.4) Edge Sample Identification and Resampling Strategy

[0186] Edge Sample Identification and Resampling Strategy adopts support vector distance discriminant function formula to improve the model's recognition ability in the fuzzy area of classification boundary and reduce misjudgment:

[0187]

[0188] (1.5) Multi-level Feature Selection

[0189] F value calculation formula:

[0190]

[0191] K-fold cross-validation accuracy calculation:

[0192]

[0193] (2) Model Building

[0194] In the task of multi-modal emotion recognition, traditional models often use only a single structure (such as MLP or LSTM) to model the fused features. Although it can achieve basic performance, when facing heterogeneous data distribution, high noise input and clinical edge individuals, its performance is often not stable or lacks generalization ability. Therefore, this paper proposes a heterogeneous multi-model combined classifier architecture (Heterogeneous Ensemble Classifier, HEC), aiming to comprehensively evaluate depression risk from multiple dimensions, and improve the overall discriminability, robustness and clinical application value of the system.

[0195] The classifier is composed of four sub-models with significant structural differences and different functions. It designs the discrimination logic from four aspects: global modeling, modal attention mechanism, state difference enhancement, and multi-task auxiliary supervision. The four sub-models work in parallel, and their outputs are fused through a trainable weighting method to obtain the final prediction probability, making the system have good adaptability when facing different sample types (such as typical depression, subclinical state, mild emotional abnormalities, etc.).

[0196] (2.1) Model structure overview

[0197] Let the fused multi-modal feature vector be:

[0198] z fusion ∈R d

[0199] where z fusion is the feature vector after multi-modal fusion, including electroencephalogram (EEG), eye movement (EM), and heart rate variability (HRV) features.

[0200] Define four parallel sub-models f1, f2, f3, f4, and output prediction probability:

[0201] pi=fi(·),i=1,2,3,4

[0202] f i : the i-th sub-model, used to output depression prediction from different angles (i = 1, 2, 3, 4)

[0203] p i : the prediction probability output by the i-th sub-model (obtained by Softmax).

[0204] Finally, fuse into depression prediction probability:

[0205] where

[0206] p(y=1|z fusion ) : represents the probability of the sample belonging to class 1 (i.e. "depression") under the condition of given fusion feature vector z fusion , which is the final output depression prediction probability.

[0207] λ i : the weight coefficient of the i-th sub-model, indicating the importance of the model in the final prediction, satisfying

[0208] (i.e. weighted average)

[0209] p i : the output probability of the i-th sub-model,

[0210] In particular:

[0211] p1: output of the standard MLP submodel

[0212] p1: output of the modal attention mechanism model

[0213] p3: output of the contrast enhancement model (using the difference of positive and negative emotional stimuli)

[0214] p4: output of the auxiliary task model

[0215] (2.2) Submodel f1: Standard MLP Model

[0216] As the most basic branch, f1 directly models the fused features in a non-linear way:

[0217] p1 = f1(z fusion ) = Softmax(W1 · z fusion + b1)

[0218] W1: weight matrix;

[0219] b1: bias vector;

[0220] Softmax(·): used to transform the output into probabilities;

[0221] f1(·): first submodel, i.e., standard multi-layer perceptron (MLP) model.

[0222] p1: predicted probability output by this model (e.g., probability of being depressed).

[0223] (2.3) Submodel f2: Modal Attention Modeling MLP

[0224] To cope with the dynamic changes in the importance of different modalities, an attention mechanism is introduced to adjust the modal weights:

[0225]

[0226] p2 = f2(z fusion ) = Softmax(W2 · z fusion + b2)

[0227] This formula is used to calculate the attention weight α i of the i-th modality. It measures the importance of different modal features z i through a scoring function with a nonlinear activation (tanh).

[0228] where:

[0229] z i : representation of the i-th modality;

[0230] w: trainable vector parameter in attention mechanism;

[0231] The numerator is the score of the current modality, and the denominator is the normalization of all modality scores, and the overall forms a Softmax structure, so that all α i Add up to 1.

[0232] z i : represents the embedding representation or feature vector of the iii-th modality (e.g. EEG, HRV or EM).

[0233] w: a trainable weight vector for calculating the importance of different modalities (attention score).

[0234] tanh(·): hyperbolic tangent function as an activation function to introduce a nonlinear transformation for input features.

[0235] exp(·): exponential function to amplify differences and form a softmax format.

[0236] α i : represents the attention weight of the i-th modality, the larger the value, the higher the importance of the modality to the final classification task.

[0237] M: the total number of modalities, which is 3 (EEG, eye movement, heart rate variability) in this task.

[0238] (2.4) Sub-model f3: contrast enhancement model

[0239] This module uses the difference between the responses to "positive stimuli" and "negative stimuli" in the emotional task as an enhanced feature modeling:

[0240]

[0241]

[0242] zf(positive): the fusion feature representation extracted by the model under positive emotional stimuli.

[0243] zf(negative): the fusion feature representation extracted by the model under negative emotional stimuli.

[0244] Δz: difference vector of the two states (negative feature minus positive feature), used to capture the response intensity of the subject to different emotional stimuli, which is an enhanced feature.

[0245] [z fusion ,Δz]: represents the splicing of the original fusion feature and the emotional difference feature, used to enhance the input expression ability of the model.

[0246] w3: weight matrix of submodel f3.

[0247] b3: bias term.

[0248] f3(·): contrast enhancement modeling module.

[0249] p3: output probability of submodel f3, considering both basic features and emotion-sensitive responses.

[0250] (2.5) Submodel f4: auxiliary task model

[0251] Submodel f4 is an auxiliary task modeling module, which aims to predict the behavioral variables (such as reaction time and score) of individuals in parallel while performing the main task (depression classification), to achieve multi-task joint learning, thereby enhancing the generalization ability and discriminative ability of the main task.

[0252] This model predicts behavioral variables (such as reaction time r and score s) in parallel and jointly with the main task:

[0253] p4 = f4 (main) (z fusion ) = Softmax(W4·z fusion +b4)

[0254]

[0255] This formula represents submodel f4 performing main task classification (predicting depression probability) on fusion feature zfusion\mathbf{z}_{fusion}zfusion.

[0256] W4,b4 are learnable parameters;

[0257] The output is the probability p4, representing the prediction probability of the sample being "depressed".

[0258] r^: predicted reaction time;

[0259] s^: predicted behavioral score;

[0260] w r ,w s and b r ,b s are trainable parameters for auxiliary tasks.

[0261] Calculate the mean square error (MSE) for the two behavioral variables respectively;

[0262] λ r ,λ s : weighting coefficients representing auxiliary loss.

[0263] (2.6) Output fusion and uncertainty modeling

[0264] The four sub-model outputs are combined by weighting:

[0265]

[0266] To improve the reliability of diagnosis, Beta distribution modeling uncertainty is added:

[0267] α = exp(W α ·z fusion ), β = exp(W β ·z fusion )

[0268]

[0269] The Beta distribution parameters α and β are used to describe the confidence of the model in the prediction results.

[0270] : the expectation of the prediction result (i.e., the average prediction probability);

[0271] Var(y): the variance of the prediction result, used to measure uncertainty, the larger the value, the more uncertain the model.

[0272] (3) Model optimization

[0273] (3.1) Curriculum learning mechanism

[0274] This method introduces a curriculum learning mechanism. This strategy controls the training order by constructing sample difficulty levels, allowing the model to learn gradually from easy to difficult. Specifically, "simple samples" with significant discriminative characteristics are preferentially input in the initial training set to help the model quickly establish preliminary classification ability; as training progresses, "complex samples" with lower discriminative or ambiguous boundaries are gradually introduced to enhance the model's judgment ability on critical samples. This method can effectively improve the convergence speed and generalization ability of the model, and avoid falling into local optimum.

[0275] Sample difficulty evaluation:

[0276]

[0277] Training step-by-step construction process:

[0278]

[0279] (3.2) Adversarial training mechanism

[0280] In addition, to improve the robustness of the model to noise and atypical data, the method introduces an adversarial training mechanism. By applying a small perturbation to the original sample to generate an adversial sample, the model learns both the real sample and its "adversarial sample" during the training process, thereby improving the recognition ability of the small perturbation. At the same time, the generated adversarial network (GAN) is used to synthesize realistic pseudo-physiological data, expand the training sample pool, effectively alleviate the sample scarcity problem, and enhance the generalization performance of the model under different conditions.

[0281] Adversarial perturbation generation:

[0282]

[0283] The loss function of the generative adversarial network:

[0284]

[0285] The content realized by the embodiment includes:

[0286] (1) Based on the dynamic interaction mechanism of adolescents and the environment, effective VR intelligent interaction scenarios and measurement indicators, according to the characteristics of the physical and mental development of adolescents, a three-dimensional interactive VR scene is designed, which involves VR scene script design, multi-sensory channel activation (vision, hearing), interaction between adolescents and key characters, and interaction between adolescents and the environment. A multi-factor mixed experiment design is used to manipulate the core influencing variables and provide a stable and controllable experimental scene.

[0287] (2) The electroencephalogram, eye movement and heart rate variability synchronous acquisition network is used to ensure that the electroencephalogram and heart rate variability data recorded during the free browsing of the emotional face of the subject are synchronized to millisecond precision, which is the basis for meaningful synchronization analysis of electroencephalogram, eye movement and heart rate variability.

[0288] (3) The signal processing method is used to process the data of the electroencephalogram, mainly for the three wave bands (alpha, beta and theta) that can better identify depression.

[0289] (4) After completing the single-mode depression recognition research based on electroencephalogram, eye movement and heart rate variability features, the single-mode electroencephalogram / eye movement / heart rate variability features are input into the denoising autoencoder to generate their respective shared representations for training the classifier, and the single-mode depression recognition result is obtained.

[0290] (5) Taking advantage of the complementarity of different modalities, the multi-modal depression recognition research based on electroencephalogram, eye movement and heart rate variability features is completed, and two modal fusion strategies (feature fusion and hidden layer fusion) are used to fuse the features of electroencephalogram, eye movement and heart rate variability data at the feature layer to improve the classification accuracy and build a more accurate depression recognition model.

[0291] (6) Complete the cross-modal depression recognition research based on EEG, eye movement, heart rate variability characteristics, explore whether the shared representation learned by the automatic encoder has strong correlation with adolescent depression, and weak correlation with the signal form of EEG / eye movement or heart rate variability.

[0292] Embodiment 2

[0293] The embodiment is a training system of an auxiliary diagnosis model for adolescent depression based on multi-modal data, comprising:

[0294] A virtual reality subsystem is configured to construct a VR intelligent interaction scene, and synchronously collect physiological data of adolescent depression and normal control subjects in the VR intelligent interaction scene, wherein the physiological data includes EEG data, heart rate variability data, and eye movement data.

[0295] A data processing subsystem is configured to perform data preprocessing and feature extraction on the EEG data, heart rate variability data, and eye movement data, to obtain EEG feature data, heart rate variability feature data, and eye movement feature data.

[0296] A single-modal depression recognition model training subsystem is configured to train a classifier based on the EEG feature data, heart rate variability feature data, and eye movement feature data, respectively, to obtain a single-modal depression recognition model, wherein the single-modal depression recognition model includes a depression recognition model based on EEG data, a depression recognition model based on heart rate variability data, and a depression recognition model based on eye movement data.

[0297] A multi-modal depression recognition model training subsystem is configured to perform feature layer fusion on the EEG feature data, heart rate variability feature data, and eye movement feature data, and then train a classifier, to obtain a multi-modal depression recognition model.

[0298] A cross-modal depression recognition model training subsystem is configured to take the EEG feature data, heart rate variability feature data, and eye movement feature data as inputs of an automatic encoder, respectively, to generate respective shared representations; during classifier training, only one kind of feature data is provided, and the other two kinds of feature data are tested on the trained classifier, to obtain a cross-modal depression recognition model.

[0299] The virtual reality subsystem includes a Biopac MP160 physiological multichannel instrument and an aSee A8 portable remote eye tracker, which are configured to acquire EEG data, eye movement data, and heart rate variability data.

[0300] Obviously, the above embodiments are merely example for clearly illustrating but not limitation to the embodiments. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments need not and can not be enumerated. The obvious changes or variations derived from the above description are still within the protection scope of the present application.

Claims

1. A method for training an auxiliary diagnosis model for adolescent depression based on multi-modal data, characterized in that, The application relates to a method for constructing a VR intelligent interaction scene, collecting physiological data of a teenager with depression and a normal control subject in the VR intelligent interaction scene, and training a classifier based on the physiological data. The method comprises the following steps: Step S1, constructing a VR intelligent interaction scene, and synchronously collecting physiological data of a teenager with depression and a normal control subject in the VR intelligent interaction scene, wherein the physiological data comprises electroencephalogram data, heart rate variability data and eye movement data; Step S2, performing data preprocessing and feature extraction on the electroencephalogram data, the heart rate variability data and the eye movement data, to obtain electroencephalogram feature data, heart rate variability feature data and eye movement feature data; The electroencephalogram data preprocessing comprises the following steps: performing 0.5-100Hz band-pass filtering on original electroencephalogram signals to remove low-frequency drift and high-frequency noise; applying a 50Hz notch filter to eliminate power supply interference; performing signal denoising through discrete wavelet transform; calculating SNR and RMSE to evaluate signal quality; and finally separating the processed signals into five frequency bands of Delta, Theta, Alpha, Beta and Gamma for power spectral density analysis; The heart rate variability data preprocessing comprises the following steps: applying 0.04-5Hz band-pass filtering to original photoplethysmogram signals to remove baseline drift and high-frequency noise; performing signal denoising through discrete wavelet transform; identifying heartbeat intervals through a peak detection algorithm; and extracting time domain and frequency domain indexes; The eye movement data preprocessing comprises the following steps: if the missing value of a data tuple in the eye movement data exceeds a set threshold, the data tuple is discarded; if the missing value of the data tuple does not exceed the set threshold, the missing value is filled with an average value or a median value; The feature extraction adopts a correlation-based feature selection method combined with a best-first search strategy; Gaussian filtering formula denoising, Gaussian noise addition for data enhancement, weighted cross-entropy loss function for sample rebalancing, support vector distance discriminant function formula for edge sample identification and resampling; defining the significance P value of the evaluation feature, removing the features with insignificant differences through difference test between the evaluation feature and a target label, and finally performing K-fold cross-validation to determine an optimal feature subset based on the comprehensive performance on the training set; Step S3, training a classifier based on the electroencephalogram feature data, the heart rate variability feature data and the eye movement feature data respectively, to obtain a single-mode depression recognition model; the single-mode depression recognition model comprises a depression recognition model based on the electroencephalogram data, a depression recognition model based on the heart rate variability data and a depression recognition model based on the eye movement data; Any feature data in the electroencephalogram feature data, the heart rate variability feature data and the eye movement feature data is taken as an input layer of an autoencoder, noise is added in the original input, a shared representation is generated through encoding, and the shared representation is taken as input data for classifier training, and then the original input is reconstructed through decoding; Step S4, performing feature layer fusion on the electroencephalogram feature data, the heart rate variability feature data and the eye movement feature data, and then training a classifier to obtain a multi-mode depression recognition model. The feature layer fusion step is: the electroencephalogram feature data, the heart rate variability feature data and the eye movement feature data are directly connected together and input into an automatic encoder to generate a shared representation; and the weight and bias of the automatic encoder are fine-tuned using an unsupervised back propagation algorithm to generate a final shared representation for actual training of a classifier; The fused multi-modal feature vector is denoted as: z fusion ∈R d wherein z fusion is the feature vector after multi-modal fusion; Four parallel sub-models f1, f2, f3 and f4 are defined to output prediction probabilities: pi=fi(·), i=1,2,3,4 wherein f i is the i-th sub-model for outputting depression prediction at different angles; p i the predicted probability output for the i-th sub-model; The fusion is a depression prediction probability: wherein where p(y = 1 | z fusion ): represents the probability of the sample belonging to class 1, which is depression, given z fusion . λ i is the weight coefficient of the i-th sub-model, satisfying p i is the output probability of the i-th sub-model, specifically: p1 is the output of the standard MLP sub-model; p2 is the output of the modality attention mechanism model; p3 is the output of the contrast enhancement model; p4 is the output of the auxiliary task model; The sub-model f1 is a standard MLP model for non-linear modeling of the fused features: p1 = f1(z fusion ) = Softmax(W1 · z fusion + b1) W1 is a weight matrix; b1 is a bias vector; Softmax(·) is used to transform the output into a probability; The sub-model f2 is a modality attention modeling MLP for modality weight adjustment: p2 = f2(z fusion ) = Softmax(W2 · z fusion + b2) wherein: z i is the representation for the i-th modality; w is a trainable vector parameter in the attention mechanism; w is a trainable weight vector for calculating the importance of different modalities; tanh(·) is a hyperbolic tangent function as an activation function to introduce a non-linear transformation for the input features; exp(·) is an exponential function for amplifying differences and forming a softmax format; alpha i represents the attention weight of the i-th modality, the larger the value, the higher the importance of the modality to the final classification task; M is the total number of modalities; The sub-model f3 is a contrast enhancement model that uses the difference between the responses to positive and negative stimuli in the emotion task as an enhanced feature modeling: wherein, zf(positive) is the fused feature representation extracted by the model under positive emotional stimuli; zf(negative) is the fused feature representation extracted by the model under negative emotional stimuli; Δz is a difference vector of the two states for capturing the response intensity of the subject to different emotional stimuli, which is an enhanced feature; [z fusion , Δz] represents the splicing of the original fusion features and the emotion difference features, used to enhance the input expression ability of the model; w3 is a weight matrix of the sub-model f3; b3 is a bias term; f3(·) is a contrast enhancement modeling module; p3 is the output probability of the sub-model f3; The sub-model f4 is an auxiliary task modeling module for predicting individual behavior variables in parallel while performing the main task: f4 = f4(main)(z fusion ) = Softmax(W4 · z fusion + b4) where W4, b4 are learnable parameters; The output is a probability p4 representing the prediction probability of the sample being depressed; r^ is the predicted reaction time; s^ is the predicted behavior score; w r ,w s and b r ,b s are trainable parameters for the auxiliary task; The mean square error MSE is calculated for the two behavior variables respectively; λ r ,λ s denotes a weighting factor for the auxiliary loss; The outputs of the four sub-models are combined by weighting: Beta distribution modeling is added to model uncertainty: a = exp(W α ·z fusion ), β = exp(W β ·z fusion ) The Beta distribution parameters a, b are used to describe the confidence of the model in the prediction result; is the expected value of the prediction result; Var(y) is the variance of the prediction result for measuring uncertainty, and the larger the value, the more uncertain the model is. In step S5, the electroencephalogram feature data, the heart rate variability feature data and the eye movement feature data are respectively input into an automatic encoder to generate respective shared representations; only one kind of feature data is provided during classifier training, and the other two kinds of feature data are tested on the trained classifier to obtain a cross-modal depression recognition model.

2. The method of claim 1, wherein, Course learning and adversarial training mechanisms are used to optimize the model.

3. The method of claim 1, wherein, The VR intelligent interaction scene adopts an A-Frame framework to build a Web VR environment, and builds a mind dialogue space through a Claude API.

Images