Depressive emotion early screening and intervention system based on cross-modal heterogeneous information

Abstract

The invention discloses a depressive emotion early screening and intervention system based on cross-modal heterogeneous information. The depressive emotion early screening and intervention system comprises a tester information registration module, a sensor test template, a resting state data acquisition module, a task state data acquisition module, a depressive emotion evaluation module, a digital intervention module, a report display module and a terminal control module. According to the method, scale, physiological data and behavioral signals are combined, and multiple scientific evaluation paradigms are constructed to perform multi-dimensional, objective and effective evaluation on the depressive emotion; the system further provides a personalized digital intervention scheme through an evaluation result, and the depressive emotion is preliminarily relieved. According to the invention, early screening and intervention of the depression emotion state can be realized, the detection efficiency is improved, and the method can be applied to various scenes such as schools, hospitals and families.

Description

[0001] The field mentioned above

[0002] The present invention relates to the field of psychological assessment and application technologies, and particularly relates to a system for early screening and intervention of depressive emotions based on cross-modal heterogeneous information Background Art

[0004] Currently, the assessment of depression mainly relies on means such as clinical symptom observation, psychological tests, and medical examinations. However, these assessment modalities are often single, and it is difficult to comprehensively and accurately reflect the psychological state of patients. Due to the singularity of depression assessment modalities and the lack of self-awareness of patients, the early detection rate of depression is generally low. When many patients have depressive symptoms, they often choose to bear them by themselves or seek non-professional help, missing the best treatment opportunity. In addition, some students, teachers, and parents lack a correct understanding of depression and it is difficult to detect the abnormal manifestations of patients in a timely manner, further exacerbating the difficulty of early screening. Moreover, there is currently a lack of digital intervention therapies for personalized intervention of depressive emotions, and a systematic multi-dimensional evaluation and intervention system is lacking to achieve efficient screening and intervention of depressive emotions Summary of the Invention

[0005] In view of the above technical problems, the present invention provides a system for early screening and intervention of depressive emotions based on cross-modal heterogeneous information. After obtaining the basic information of the tester, it collects multi-modal behavioral and physiological signals such as electroencephalogram, electrocardiogram, voice, and facial expressions, analyzes the mapping indicators between different modal signals and depressive disorders, and on the basis of traditional scale assessment, introduces various depressive evaluation tasks such as voice, pictures, videos, etc., to construct a richer and more objective depressive evaluation paradigm. It constructs an early depressive disorder evaluation system based on multi-modal information, formulates a personalized digital intervention plan according to the evaluation results, realizes multi-dimensional evaluation and intervention of early depressive disorders, and makes up for the deficiencies of existing technologies

[0006] In order to achieve the above object, the present invention provides the following technical solutions

[0007] On the one hand, the present invention provides a system for early screening and intervention of depressive emotions based on cross-modal heterogeneous information, which is characterized by specifically including the following modules

[0008] A tester information registration module, which is connected to the terminal control module and is used for collecting and inputting basic information after the tester enters the system

[0009] Further, the basic information includes demographic information, history of mental illness, and information on events affecting emotional state, specifically including: name, gender, age, height, weight, whether coffee, tea, alcohol or other stimulating beverages were consumed within 24 hours, whether there was strenuous exercise within 24 hours, whether there is a previous history of mental illness, whether there is a previous history of brain disease, and whether there is a previous history of cardiovascular disease.

[0010] Sensor test module, which is connected to the terminal control module and is used to guide the tester to correctly wear, connect and test the physiological and behavioral data collection devices;

[0011] Further, the data collection device test includes electroencephalogram device test, electrocardiogram device test, voice device test and video device test.

[0012] Resting state data collection module, which is connected to the terminal control module and is used to collect physiological data in a 3-minute resting state. The resting state includes the state where the tester is required to sit still, close their eyes, and breathe evenly. The physiological data includes electroencephalogram signals and electrocardiogram signals, and an adaptive filtering algorithm is used to preprocess the signals;

[0013] Task state data collection module, which is connected to the terminal control module and designs a combination of multiple task paradigms such as scales, videos, questions and answers, reading aloud, and pictures to collect cross-modal heterogeneous data under different tasks, including scale information, physiological information, and behavioral information. The physiological information includes electroencephalogram and electrocardiogram signals, and the behavioral information includes voice and facial expression signals;

[0014] Further, the scale task selects the Patient Health Questionnaire-9 (PHQ-9); PHQ-9 is a simple and effective self-rating scale for depressive disorders based on the DSM-IV (Diagnostic and Statistical Manual of Mental Disorders developed by the American Psychiatric Association) diagnostic criteria. It contains 9 items and has good reliability and validity in both depressive diagnosis assistance and symptom severity assessment.

[0015] The video task is selected from a clip of "Lost in Thailand", and the video duration is 240 seconds. In this task, the participants are required to maintain a stable sitting position and try not to shake their heads as much as possible so that complete facial expressions can be collected;

[0016] The picture task is selected from the Chinese Facial Affective Picture System (CFAPS), which contains 9 pictures (3 positive valence, three neutral valence, three negative valence). The pictures are shown in the order of positive, neutral, negative, positive, neutral... alternately. Each picture is shown for 3 seconds, and there is a 3-second rest between every two pictures to eliminate the influence of the previous picture;

[0017] The reading task is selected from a passage of the prose "The North Wind and the Sun". The participants read aloud in front of the system according to the text displayed on the screen without being affected by others.

[0018] The Q&A task is selected from questions closely related to teenagers, including four questions about good friends, studies, the relationship with parents, and recent happy experiences, which can closely reflect the true feelings of teenagers. During the entire recording process, the participants answer the questions displayed on the screen independently without being disturbed by others.

[0019] Furthermore, the order of different tasks is scientifically adjusted, and questionnaire questions are designed between tasks to eliminate the mutual influence between tasks. The system can also adjust the content according to the responses of the tested person to ensure the personalization and accuracy of the evaluation. Through a scientific and reasonable task paradigm, the independence of different task states can be guaranteed, the influence of the previous question can be eliminated, and during the entire experiment, multi-dimensional vital sign signals such as the electroencephalogram, electrocardiogram, voice, and facial expression of the participants are synchronously recorded.

[0020] A depressive mood assessment module, which is connected to the terminal control module, and based on unimodal signal analysis technology and cross-modal data fusion technology, combined with standardized scale data, conducts evaluations of depressive mood in different information dimensions and multi-dimensions;

[0021] Furthermore, the depressive mood assessment module collects the electroencephalogram, electrocardiogram, voice, and facial expression signals of the tester, conducts preprocessing analysis on the signals, and obtains signal features through feature extraction, including electrocardiogram features, electroencephalogram features, cardiorespiratory coupling features, voice features, and facial expression features. Then, a correlation analysis is conducted between the signal features and the depressive state to obtain highly correlated features with depressive characteristics, and then input into a unimodal depressive assessment model to obtain a four-dimensional depressive state assessment index. Then, the cross-modal signals are multi-modally fused and input into a multi-modal easy-to-recognize model to achieve multi-modal depressive assessment, and thus complete the depressive mood assessment under a single dimension and under multi-dimensional information fusion.

[0022] A digital intervention module, which is connected to the terminal control module, is used for the system to generate a personalized intervention plan according to the evaluation results, including meditation intervention, music intervention, video intervention, etc.; the system will recommend the most suitable intervention method according to information such as the age, gender, and hobbies of the tester. For example, teenage testers may be more inclined to video intervention, while adult testers may be more inclined to mindfulness meditation intervention.

[0023] Further, after the intervention ends, the tester conducts resting-state signal acquisition again. The system analyzes the intervention effect by comparing the signal characteristics and emotion assessment results before and after the intervention, and generates a personalized intervention report for the tester to guide the adjustment of the subsequent intervention plan.

[0024] A report display module, which is connected to the terminal control module and is used to synthesize and display the tester's personal file, including the tester's basic demographic information, scale scores, analysis results of depressive emotions under different data dimensions, the tester's comprehensive depressive emotion score, and the tester's intervention record.

[0025] Further, display the tester's basic statistical information, scale scores, analysis results of depressive emotions under different data dimensions, the tester's comprehensive depressive emotion score, and the tester's intervention record.

[0026] A terminal control module; the terminal control module is connected to the tester information registration module, the sensor measurement module, the resting-state data acquisition module, the task-state data acquisition module, the depressive emotion assessment module, the digital intervention module, and the report display module, and is used to control the normal operation of each module through the system terminal.

[0027] On the other hand, the present invention provides a depressive emotion screening and intervention system based on cross-modal heterogeneous information, specifically including the following steps:

[0028] S1: The tester enters the adolescent depressive emotion screening and intervention system and completes the tester's basic information, including name, gender, age, height, weight, whether coffee, tea, or alcoholic beverages such as alcohol are consumed within 24 hours, whether there is strenuous exercise within 24 hours, whether there is a previous mental illness history, whether there is a previous brain illness history, and whether there is a previous cardiovascular illness history.

[0029] S2: The tester wears electroencephalogram and electrocardiogram signal acquisition sensors according to the system interface prompts, adjusts the sitting posture, and conducts sensor signal tests, including electroencephalogram signal tests, electrocardiogram signal tests, voice signal tests, and facial expression signal tests.

[0030] S3: The tester maintains a closed-eye resting state according to the system voice and text prompts to obtain resting-state physiological information, including electroencephalogram and electrocardiogram signals.

[0031] S4: The tester completes a series of set task-state questions according to the system prompts to obtain multi-modal heterogeneous information under different task states.

[0032] Further, the task-state questions include:

[0033] For the picture task, the system will sequentially display three pictures in different states: positive, neutral, and negative, with a 3-second interval between two pictures. After the last picture is displayed, the system will prompt "The picture playback has been completed. Please click the next question.";

[0034] For the reading task, the tester clicks "Start Recording", and the system broadcasts "Voice recording has started". After the reading is completed, click "End Recording", and the voice broadcasts "Voice recording has ended. Please click the next question.";

[0035] For the video task, the system will play a video. The tester watches the video. After the video is played, a voice prompt will be given: "The video playback has been completed. Please click the next question.";

[0036] For the Q&A task, the tester reads the questions displayed on the system interface, including questions closely related to teenagers such as good friends, studies, and relationships with parents. After a little thought, click "Start Recording", and the voice prompt is "Voice recording has started", and then start answering. After the answer is completed, click "End Recording", and the voice prompt is "Voice recording has ended. Please click the next question."

[0037] Scale task: The scale selected is the Patient Health Questionnaire-9 (PHQ-9). The scale questions are designed to appear among the first four theme task questions to ensure the mutual independence of the tasks.

[0038] S5: The system processes and analyzes information in different dimensions, combines multi-modal data fusion technology and standardized scales to evaluate different information dimensions and the overall depressive mood state.

[0039] Furthermore, the evaluation specifically includes the following stages

[0040] In the first stage, for single-dimensional information, preprocessing such as normalization and filtering of the information is performed to extract electroencephalogram (EEG) features, electrocardiogram (ECG) features, voice features, and facial expression features related to depressive mood. Then, feature correlation analysis is carried out to evaluate features highly correlated with the depressive state. Combining traditional machine learning and deep learning models, different depressive states are identified. The EEG features include EEG frequency band indicators and non-linear features. The ECG features include heart rate variability indicators and non-linear indicators. The voice features include fundamental frequency, energy, and spectral features. The facial expression features include basic Action Unit (AU) features of the face. The depressive states include mild, moderate, and severe depressive states.

[0041] In the second stage, for the multi-dimensional information such as EEG, ECG, voice, and facial expressions collected, a cross-attention mechanism is introduced to fuse heterogeneous information, integrate depressive mood-related information across modalities, and maintain the consistency and complementarity between information. The fused feature matrix is where \(v\) j is the eigenvector of mode \(j\), and \(\alpha\) ij is the attention weight between modes. Then, based on the fused feature matrix \(H\), the multi-modal depression assessment model is used to evaluate the depressive mood. \(y\) multi = \(g(H)\), where \(g\) is the multi-modal depression assessment model, and then the cross-modal information is analyzed and identified to achieve accurate assessment of depressive mood based on multi-modal information.

[0042] S6: According to the evaluation results, the system gives a personalized intervention plan through the digital intervention module, including mindfulness meditation intervention, music intervention, video intervention, etc.

[0043] S7: After the intervention, the tester conducts a resting state evaluation again to investigate the effects of different intervention methods on the depressive mood of adolescents, compare the signal characteristics and mood evaluation results before and after the intervention, and generate a personalized intervention report for the tester to guide the adjustment of the subsequent intervention plan.

[0044] S8: The report display module synthesizes a personal information file, displays the tester's demographic information, past medical history, factors interfering with emotions, etc., the mood analysis results of different signal dimensions, including signal display, signal-related eigenvalue results, and single-dimensional depressive mood scores, as well as multi-modal depressive mood scores, and the tester's intervention method records. The tester can save the analysis results locally.

[0045] Compared with the prior art, the advantages of the present invention are:

[0046] The present invention provides a depressive mood screening and intervention system based on cross-modal heterogeneous information, which realizes a systematic process for depressive mood screening and intervention, can effectively assist doctors in psychological counseling, and alleviates the problem of low manual efficiency. It can be used for the universal screening of depressive mood. The system innovates the multi-modal dimension in the field of mood assessment, comprehensively analyzes the individual's early depressive state from the dual perspectives of psychology and behavior, and to a certain extent solves the drawback of strong subjectivity in depressive assessment, and more objectively and comprehensively evaluates the psychological state of the tester. The system sets a systematic, scientific, and procedural evaluation paradigm, and the scenario-based task method effectively improves the tester's evaluation cooperation degree, so as to expand the applicable scenarios of the system, alleviate the burden of operators, and facilitate large-scale depressive mood screening and intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is the structural block diagram of the depressive mood screening and intervention system based on cross-modal heterogeneous information of the present invention.

[0048] Figure 2 is the flowchart of the sensor test module of the depressive mood screening and intervention system based on cross-modal heterogeneous information of the present invention.

[0049] Figure 3 It is the flowchart of the task-state signal acquisition module of the depression emotion screening and intervention system based on cross-modal heterogeneous information of the present invention.

[0050] Figure 4 It is the flowchart of the depression emotion assessment module of the depression emotion screening and intervention system based on cross-modal heterogeneous information of the present invention.

[0051] Figure 5 It is the display diagram of the system equipment and interface of the present invention.

[0052] Figure 6 It is the framework flowchart of the system of the present invention. Specific implementation method

[0054] Next, in combination with the accompanying drawings in the embodiments of the present invention, the technical solutions of the present invention will be described completely and in detail. Obviously, the described embodiments are only partial embodiments of the present invention, not all embodiments.

[0055] As Figure 1 shown, the depression emotion screening and intervention system based on cross-modal heterogeneous information provided by the embodiments of the present invention includes a tester information registration module, a sensor test module, a resting-state data acquisition module, a task-state data acquisition module, a depression emotion assessment module, a digital intervention module, and a display module.

[0056] The tester information registration module is used for collecting and inputting tester information, including name, gender, age, height, weight, whether stimulants such as coffee, tea, or alcohol are consumed within 24 hours, whether there is strenuous exercise within 24 hours, whether there is a previous mental illness history, whether there is a previous brain illness history, whether there is a previous cardiovascular illness history, etc.

[0057] The sensor test module, please refer to Figure 2 , is used to guide the tester to correctly connect and calibrate the physiological and behavioral data acquisition devices. The acquisition devices involve electroencephalogram acquisition sensors, electrocardiogram acquisition sensors, voice devices, and facial expression acquisition devices. After the devices are successfully connected, the signals are further calibrated;

[0058] The resting-state data acquisition module is used for collecting the physiological data of students in the resting state, including electroencephalogram signals (EEG) and electrocardiogram signals (ECG). During the resting-state data acquisition process, it is required that the tester personnel sit still, close their eyes, and breathe evenly, and maintain a three-minute resting state;

[0059] The task-state data acquisition module, please refer to Figure 3, including scales, videos, questions and answers, reading aloud, and pictures, and collecting cross-modal heterogeneous data under different tasks, including scale information, physiological information, and behavioral information. The physiological information includes EEG and ECG signals, and the behavioral signals include voice and facial expression signals;

[0060] Furthermore, the scale task is selected from Patient Health Questionnaire-9 (PHQ-9); PHQ-9 is a simple and effective self-rating scale for depressive disorders based on the diagnostic criteria of DSM-IV (Diagnostic and Statistical Manual of Mental Disorders developed by the American Psychiatric Association), which contains a total of 9 items and has good reliability and validity in assisting depression diagnosis and assessing symptom severity.

[0061] The video task is of positive valence and is selected from a clip of Lost in Thailand. The video duration is 240 seconds. In this task, participants are required to maintain a stable sitting posture and try not to shake their heads so that complete facial expressions can be collected.

[0062] The picture task was selected from the Chinese Facial Affective Picture System (CFAPS), which included 9 pictures (3 positive valences, 3 neutral valences, and 3 negative valences). The pictures were displayed in the order of positive, neutral, negative, positive, neutral, etc., alternately. Each picture was displayed for 3 seconds, and there was a rest period of 3 seconds between every two pictures to eliminate the influence of the previous picture.

[0063] The reading task reads aloud texts selected from the paragraph of "The North Wind and the Sun";

[0064] The question-and-answer task is selected from issues closely related to teenagers, including four questions about good friends, studies, relationships with parents, and recent happy experiences, which can closely reflect the true emotions of teenagers. During the entire recording process, participants independently answer the questions displayed on the screen without being disturbed by others.

[0065] Furthermore, the order of different tasks is scientifically adjusted, and questionnaire questions are designed between each task to eliminate the mutual influence between tasks. The system can also adjust the content according to the response of the test subject to ensure the personalization and accuracy of the evaluation. Through a scientific and reasonable task paradigm, the independence of different task states can be guaranteed, and the influence of the previous question can be eliminated. During the entire experiment, the participants' multi-dimensional vital signs such as forehead EEG, ECG, voice, and facial expressions are synchronously recorded.

[0066] Depression Assessment Module, see Figure 4 , based on single-modal evaluation technology and cross-modal data fusion technology, combined with standardized scale data, different information dimensions and multi-dimensional evaluation of adolescent depression are conducted;

[0067] Further, for single - dimensional information, pre - process the information by normalizing and filtering it, extract electroencephalogram (EEG) features, electrocardiogram (ECG) features, speech features, and facial expression features related to depressive mood, and then perform feature correlation analysis to evaluate features highly correlated with the depressive state. Combine traditional machine learning and deep learning models to identify different depressive states. The EEG features include EEG frequency band indicators and non - linear features. The ECG features include heart rate variability indicators and non - linear indicators. The speech features include fundamental frequency, energy, and spectral features. The facial expression features include basic Action Unit (AU) features of the face. The depressive states include mild, moderate, and severe depressive states.

[0068] In the second stage, for the multi - dimensional information such as EEG, ECG, speech, and facial expressions collected, introduce a cross - attention mechanism to fuse heterogeneous information, integrate depressive - mood - related information across modalities, and maintain the consistency and complementarity between information. The fused feature matrix is where v j is the feature vector of modality j, and α ij is the attention weight between modalities. Then, based on the fused feature matrix H, perform a depressive - mood assessment through a multi - modal depressive assessment model. y multi = g(H), where g is the multi - modal depressive assessment model. Further analyze and identify cross - modal information to achieve accurate assessment of depressive mood based on multi - modal information.

[0069] The digital intervention module, which is connected to the terminal control module, is used for the system to generate personalized intervention plans according to the assessment results, including meditation intervention, music intervention, video intervention, etc.; the system will recommend the most suitable intervention method according to information such as the age, gender, and hobbies of the tester. For example, adolescent testers may be more inclined to video intervention, while adult testers may be more inclined to mindfulness meditation intervention. After the intervention, the tester performs resting - state signal acquisition again. The system analyzes the intervention effect by comparing the signal features and mood assessment results before and after the intervention, and generates a personalized intervention report for the tester to guide the adjustment of subsequent intervention plans.

[0070] The report display module, which is connected to the terminal control module, is used to synthesize and display the personal profile of the tester, including the tester's basic demographic information, scale scores, depressive - mood analysis results under different data dimensions, the tester's comprehensive depressive - mood score, and the tester's intervention records;

[0071] The terminal control module; the terminal control module is connected to the tester information registration module, sensor measurement module, resting - state data acquisition module, task - state data acquisition module, depressive - mood assessment module, digital intervention module, and report display module, and is used to control the normal operation of each module through the system terminal. Figure 5It shows the interfaces of various modules of the system. The interface design is simple and intuitive, enhancing the convenience and interactivity during the evaluation process.

[0072] Please refer to Figure 6 , a depression emotion early screening and intervention system based on cross-modal heterogeneous information, including the following steps.

[0073] The tester enters the adolescent depression emotion screening and intervention system and completes the basic information of the tester, including name, gender, age, height, weight, whether drinking stimulating beverages such as coffee, tea or alcohol within 24 hours, whether having strenuous exercise within 24 hours, whether having a previous mental illness history, whether having a previous brain illness history, and whether having a previous cardiovascular illness history.

[0074] The tester wears electroencephalogram and electrocardiogram signal acquisition sensors according to the system interface prompts, adjusts the sitting posture, and conducts sensor signal tests, including electroencephalogram signal test, electrocardiogram signal test, voice signal test, and facial expression signal test.

[0075] The tester conducts cross-modal signal acquisition according to the system voice and text prompts, including resting state signal acquisition and task state signal acquisition. Among them, the resting state signal acquisition requires maintaining a closed-eye resting state to obtain resting state physiological information, including electroencephalogram and electrocardiogram signals; the task state signal acquisition requires the tester to complete a series of set task state questions according to the system prompts to obtain multi-modal heterogeneous information under different task states. The task states include scale task, video task, question-and-answer task, reading task, and picture task. The scale questions are designed to appear among the four theme task questions to ensure the mutual independence of tasks.

[0076] The system processes and analyzes information in different dimensions, combines single-modal signal analysis and multi-modal data fusion technology to evaluate the depression emotion states in different information dimensions and overall.

[0077] For single-dimensional information, preprocess the information by normalizing and filtering, extract electroencephalogram features, electrocardiogram features, voice features, and facial expression features related to depression emotion, then conduct feature correlation analysis to evaluate the features highly correlated with the depression state, and combine traditional machine learning and deep learning models to identify different depression states.

[0078] For multi-dimensional information such as electroencephalogram, electrocardiogram, voice, and facial expression collected, introduce a cross-attention mechanism to fuse heterogeneous information, integrate the depression emotion-related information between cross-modalities, and maintain the consistency and complementarity between information. The fused feature matrix is used for depression emotion assessment through a multi-modal depression assessment model, and then the cross-modal information is analyzed and identified to achieve accurate assessment of depression emotion based on multi-modal information.

[0079] Based on the evaluation results, the system gives a personalized intervention plan through the digital intervention module, including mindfulness meditation intervention, music intervention, and video intervention;

[0080] After the intervention, the tester conducts a resting state evaluation again to investigate the effects of different intervention methods on depressive mood, compare the signal characteristics and mood evaluation results before and after the intervention, and generate a personalized intervention report for the tester to guide the adjustment of the subsequent intervention plan.

[0081] The report display module synthesizes a personal information profile, showing the tester's demographic information, past medical history, factors interfering with mood, etc., the emotional analysis results of different signal dimensions, including signal display, signal-related eigenvalue results, and single-dimensional depressive mood scores, as well as multi-modal depressive mood scores, and the tester's intervention method record. The tester can save the analysis results locally.

[0082] The system is a portable device with a small size and easy to carry. It can be evaluated at any time whether at home, in the hospital or clinic, reducing time and transportation costs, and greatly improving the practicality and application scope of the system.

[0083] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A depression emotion early screening and intervention system based on cross-modal heterogeneous information, characterized in that, It includes the following modules: A tester information registration module for collecting and inputting tester information; A sensor testing module for guiding the tester to correctly connect and calibrate physiological and behavioral data collection devices; A resting-state data collection module for collecting physiological data of the tester in a resting state, including electroencephalogram (EEG) signals and electrocardiogram (ECG) signals; A task-state data collection module that designs multiple task paradigms combining scales, videos, questions and answers, reading aloud, and pictures to collect cross-modal heterogeneous data in different task paradigms, including scale information, physiological information, and behavioral information. The physiological information includes EEG and ECG signals, and the behavioral information includes voice and facial expression data; A depressive mood assessment module that conducts multi-dimensional assessments of the depressive mood state based on unimodal assessment techniques and cross-modal data fusion techniques, combined with standardized scale data; A digital intervention module for generating personalized intervention plans according to the assessment results, including meditation intervention, music intervention, and video intervention; A report display module for displaying the tester's personal profile; A terminal control module; It is used to connect to each module and control each module to work properly through the terminal.

2. The early screening and intervention system for depressive mood based on cross-modal heterogeneous information according to claim 1, wherein The tester information registration module is used to collect and input the tester's demographic information, history of mental illness, and event information affecting the emotional state.

3. The early screening and intervention system for depressive mood based on cross-modal heterogeneous information according to claim 1, characterized in that, The resting-state signal collection module is used to collect EEG and ECG signals in the eyes-closed resting state and perform denoising processing on the signals using an adaptive preprocessing algorithm.

4. The early screening and intervention system for depressive mood based on cross-modal heterogeneous information according to claim 1, characterized in that, The content of the specific task state of the task-state data collection module includes: Scale task: The scale selected by the system is the Patient Health Questionnaire 9; Video task: The video clip has a positive valence. In this task, the participant is required to maintain a stable sitting posture and try not to shake their head so that a complete facial expression can be collected; Picture task: It contains 9 pictures, among which, 3 have positive valence, 3 have neutral valence, and 3 have negative valence; The pictures are shown as positive, neutral, and negative in turn and alternately, each picture is shown for 3 seconds, and there is a 3-second rest between every two pictures to eliminate the influence of the previous picture; The experimental pictures are selected from the Chinese Facial Affective Picture System; Question and answer task: A total of 4 questions are selected, covering four aspects: good friends, studies, relationship with parents, and the happiest recent experience. The questions are closely related to teenagers and can closely reflect the true feelings of teenagers; During the entire recording process, the participant answers the questions displayed on the screen independently without being interfered by others; Reading aloud task: The text is selected from an article paragraph. The participant reads aloud in front of the system according to the text displayed on the screen without being affected by others; To ensure the independence of different task states, by adjusting the task order and designing questionnaire questions between tasks, the influence of the previous question is eliminated. During the entire experiment, multi-dimensional vital signs signals of the participant's frontal EEG, ECG, voice, and facial expressions are synchronously recorded.

5. The early screening and intervention system for depressive mood based on cross-modal heterogeneous information according to claim 1, characterized in that The depression emotion assessment module collects the electroencephalogram (EEG), electrocardiogram (ECG), voice, and facial expression signals of the tester, preprocesses and analyzes the signals, and obtains the signal features of ECG, EEG, cardiopulmonary coupling, voice, and facial expression through feature extraction; conducts a correlation analysis between the signal features and the depression state to obtain highly correlated features with depressive manifestations. Then, it inputs into the single-modal depression assessment model to obtain the four-dimensional depression state assessment index; then, conducts multi-modal fusion on the cross-modal signals and inputs them into the multi-modal easy-to-recognize model to achieve multi-modal depression assessment, and further completes the depression emotion assessment under a single dimension and under multi-dimensional information fusion.

6. The early screening and intervention system for depressive mood based on cross-modal heterogeneous information according to claim 1, characterized in that, The digital intervention module includes mindfulness meditation intervention, music intervention, and video intervention; after the intervention, the tester conducts resting state signal collection again.

7. The early screening and intervention system for depressive mood based on cross-modal heterogeneous information according to claim 1, wherein The report display module is used to synthesize and display the personal file of the tester, including the basic demographic information of the tester, scale scores, depression emotion analysis results under different data dimensions, the comprehensive depression emotion score of the tester, and the intervention record of the tester.

8. A depression emotion early screening and intervention system based on cross-modal heterogeneous information according to any one of claims 1-7, characterized in that The intervention method of the system includes the following steps: S1: The tester logs in to the depression emotion screening and intervention system and completes the basic information of the tester. S2: The tester adjusts the sitting posture according to the system prompt, wears the EEG and ECG signal collection sensors, and conducts sensor signal tests and calibrations one by one. S3: The tester maintains a closed-eye resting state according to the system prompt to obtain resting state physiological information. S4: The tester completes multiple scale questions, question-and-answer questions, picture questions, reading questions, and video question tasks according to the system questions to obtain multi-modal heterogeneous information under different task states. S5: The system processes and analyzes information in different dimensions, combines multi-modal data fusion technology and standardized scales to evaluate the depression emotion state in different information dimensions and overall. S6: The system gives a personalized intervention plan through the digital intervention module according to the evaluation results. The intervention plan includes meditation intervention, music intervention, and video intervention. S7: After the intervention, the tester conducts resting state signal collection again; the system analyzes the intervention effect by comparing the signal features and emotion evaluation results before and after the intervention, and generates a personalized intervention report for the tester to guide the adjustment of the subsequent intervention plan. S8: The report display module synthesizes the personal information file, displays the basic information of the tester and the emotion analysis results in different signal dimensions, and the tester can save the analysis results locally.

9. The early screening and intervention system for depressive mood based on cross-modal heterogeneous information according to claim 8, characterized in that, In step S4, the system questions specifically include: Picture task: The system will sequentially display three pictures in different states of positive, neutral, and negative, with a 3-second interval between two pictures; after the display of the last picture is completed, the system will prompt "The picture playback has been completed, please click the next question". Reading task: The tester clicks "Start Recording", the system broadcasts "The voice recording has started", and after the reading is completed, clicks "End Recording", and the voice broadcasts "The voice recording has ended, please click the next question". Video task: The system will play a video, and the tester watches the video. After the video playback is completed, the voice prompts "The video playback has been completed, please click the next question". In the Q&A task, the tester reads the questions displayed on the system interface, including questions closely related to teenagers such as good friends, academics, and the relationship with parents. After a little thought, the tester clicks "Start Recording", and a voice prompt "Voice recording has started" appears. Then the tester starts answering. After the answer is completed, the tester clicks "End Recording", and a voice prompt "Voice recording has ended. Please click the next question" appears; In the scale task, the selected scale is the Patient Health Questionnaire 9. The scale questions are designed to appear among the first four theme task questions to ensure the mutual independence of the tasks.

10. A depression emotion early screening and intervention system based on cross-modal heterogeneous information according to claim 8, characterized in that, In step S5, the evaluation specifically includes the following stages: In the first stage, for single-dimensional information, preprocessing such as normalization and filtering of the information is performed to extract electroencephalogram (EEG) features, electrocardiogram (ECG) features, voice features, and facial expression features related to depressive mood. Then, feature correlation analysis is carried out to evaluate features highly correlated with the depressive state. Combining traditional machine learning and deep learning models, different depressive states are identified. The EEG features include EEG frequency band indicators and nonlinear features. The ECG features include heart rate variability indicators and nonlinear indicators. The voice features include fundamental frequency, energy, and spectral features. The facial expression features include basic Action Unit (AU) features of the face. The depressive states include mild, moderate, and severe depressive states; In the second stage, for the multi-dimensional information of EEG, ECG, voice, and facial expressions collected, a cross-attention mechanism is introduced to fuse heterogeneous information, integrate the depression emotion-related information between cross-modalities, and maintain the consistency and complementarity between information; the fused feature matrix is where v j is the feature vector of modality j, and α ij is the attention weight between modalities; then, based on the fused feature matrix H, depression emotion assessment is performed through a multi-modal depression assessment model; y multi = g(H), where g is the multi-modal depression assessment model; furthermore, the cross-modal information is analyzed and identified to achieve accurate assessment of depression emotion based on multi-modal information.

Images