AI mental health assessment and grading early warning system and method based on multi-modal fusion

Abstract

The application relates to the technical field of mental health assessment, in particular to an AI mental health assessment and grading early warning system based on multi-modal fusion, which comprises an evaluation terminal module, a camera module and a wearable device module; the evaluation terminal module is used for mental health self-evaluation of a measured object through a PHQ-9 / GAD-7 scale, and determines a mental health grade result and grading early warning of the measured object; the camera module is used for collecting expression data of the measured object in the mental health self-evaluation process of the measured object, and obtaining an emotion recognition result of the measured object; and the wearable device module is used for collecting physiological data of the measured object. On the basis of scale data, objective data such as physiological data and expression data are added, which are part of data indexes that are difficult to conceal in human emotional responses, such as physiology and behavior, the subjective concealed components of the test are effectively reduced, multi-modal data combining subjectivity and objectivity are used for mental health assessment, and the assessment accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mental health assessment, in particular to an AI mental health assessment and grading early warning system and method based on multi-modal data fusion. BACKGROUND

[0002] Tumor patients are prone to have mental health problems such as anxiety, depression, and emotional disorders, and have a huge demand for psychological assessment. The traditional psychological assessment of patients in the tumor ward mainly relies on nurses using PHQ-9 / GAD-7 scale, which has certain limitations, mainly in: 1, the data is provided by the patient himself, and the patient may deliberately conceal his own psychological problems, resulting in subjective scale data, and the psychological health assessment result is not reliable, and the manual collection and analysis efficiency is not high; 2, there is currently a lack of unified psychological intervention measures or processes, and the psychological nursing level of different nurses varies greatly due to the emphasis of the department and their own life experience; 3, patients repeatedly hospitalized, but lack of continuous attention to patients with moderate to severe psychological problems.

[0003] In the prior art, the psychological health is assessed by collecting and analyzing scale data through artificial intelligence, which can solve the defect of low efficiency of the traditional method, but due to the subjectivity of the scale data, the evaluation process still has subjectivity, and only using scale data of a single category for evaluation, the evaluation accuracy is difficult to guarantee. SUMMARY

[0004] The purpose of the present application is to provide an AI mental health assessment and grading early warning system and method based on multi-modal fusion, to solve the technical problems that due to the subjectivity of the scale data, the evaluation process still has subjectivity, and only using scale data of a single category for evaluation, the evaluation accuracy is difficult to guarantee.

[0005] To solve the above technical problems, the present application specifically provides the following technical solutions:

[0006] An AI mental health assessment and grading early warning system based on multi-modal fusion, comprising:

[0007] An evaluation terminal module for a measured object to perform a mental health self-evaluation through a PHQ-9 / GAD-7 scale, and to determine a mental health level result of the measured object according to a pre-established multi-modal mental health assessment model through the mental health self-evaluation result, an emotional recognition result and physiological data, and to display the mental health level result and health guidance to the measured object, and to perform a grading early warning to a medical object according to the mental health level result;

[0008] The camera module is used for collecting expression data of the measured object in the mental health self-evaluation process of the measured object, and emotion classification of the measured object is performed according to a pre-established emotion recognition model to obtain an emotion recognition result of the measured object.

[0009] The wearable device module is used for collecting physiological data of the measured object in the mental health self-evaluation process of the measured object.

[0010] The synchronism optimization module is arranged between the evaluation terminal module and the camera module, and is used for maintaining the synchronism between the multi-modal mental health evaluation model and the emotion recognition model in an optimization manner.

[0011] As a preferred scheme of the present application, the camera module comprises a camera and a data processor.

[0012] The camera is used for collecting expression data of the measured object in the mental health self-evaluation process of the measured object.

[0013] The data processor is used for performing emotion classification of the measured object according to the expression data of the measured object by using a pre-established emotion recognition model to obtain an emotion recognition result of the measured object.

[0014] The emotion recognition model is a classification network for inputting an expression image and outputting an emotion category, and a structure expression of the emotion recognition model is as follows:

[0015] ;

[0016] In the formula, is an emotion category output by the emotion recognition model, is an expression image, is a classification network.

[0017] The emotion category comprises anger, disgust, fear, happy, normal, sad, and surprised.

[0018] As a preferred scheme of the present application, the method for performing emotion classification of the measured object by the data processor according to the expression data by using the pre-established emotion recognition model comprises the following steps.

[0019] All expression images collected by the measured object in the mental health self-evaluation process are subjected to clustering processing by using a K-means algorithm to obtain an expression image clustering result.

[0020] An expression image located at a center point in a cluster with the largest number of images in the expression image clustering result is input into the emotion recognition model to obtain an emotion category of the measured object.

[0021] The emotion category of the tested object is used as the emotion recognition result.

[0022] As a preferred embodiment of the present invention, the wearable device module includes a smartwatch, a wristband, and a chest strap;

[0023] The physiological data collected by the wearable device module includes heart rate (HR), blood oxygen (SpO2), respiratory rate (RR), and heart rate variability (HRV).

[0024] As a preferred embodiment of the present invention, the multimodal mental health assessment model is a classifier that takes as input the self-assessment results of mental health, emotion recognition results, and physiological data, and outputs the level of mental health.

[0025] The structural expression of the multimodal mental health assessment model is as follows:

[0026] ;

[0027] In the formula, The mental health level output by the multimodal mental health assessment model. For the self-assessment results of mental health, According to the emotion category, This refers to all physiological data collected from the subjects during the self-assessment process of mental health. For classifiers.

[0028] As a preferred embodiment of the present invention, the method by which the synchronization optimization module maintains synchronization between the multimodal mental health assessment model and the emotion recognition model includes:

[0029] Based on the input and output of the multimodal mental health assessment model, a convolutional neural network is established, taking the output of the multimodal mental health assessment model as input and the emotion recognition result as output. The structural expression of the convolutional neural network is as follows:

[0030] ;

[0031] In the formula, The emotion recognition result output by the convolutional neural network. The mental health level output by the multimodal mental health assessment model is represented by CNN, which stands for Convolutional Neural Network.

[0032] by and The differences between them serve as a synchronization target between multimodal mental health assessment models and emotion recognition models. , It is an L2 norm;

[0033] When the goal is synchronization If the value exceeds the preset threshold, the current multimodal mental health assessment model and emotion recognition model will be retrained to a synchronized state with the synchronization target as the optimization objective.

[0034] The retraining priority of the current multimodal mental health assessment model and emotion recognition model is associated with their model loss functions, and the priority is as follows:

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] The loss function for retraining is: ;

[0040] In the formula, Prioritizing the training of multimodal mental health assessment models The training priority for emotion recognition models, The loss function for the multimodal mental health assessment model. Let be the loss function of the emotion recognition model. For the true value of the emotion recognition result, For the true value of mental health level, The loss function for retraining;

[0041] When the goal is synchronization If the threshold is less than or equal to the preset threshold, the current multimodal mental health assessment model and emotion recognition model will remain unchanged.

[0042] As a preferred embodiment of the present invention, the mental health level includes healthy, mild, moderate and severe.

[0043] As a preferred embodiment of the present invention, the assessment terminal module sends early warning information to the medical staff when the mental health level is moderate or severe.

[0044] As a preferred embodiment of the present invention, the assessment terminal module records each psychological health level result of the tested subject and analyzes the psychological health trend of the tested subject based on multiple consecutive psychological health level results.

[0045] If the mental health of the tested subjects shows a deteriorating trend, a warning message will be sent to the medical staff.

[0046] As a preferred embodiment of the present invention, the present invention provides an AI-based mental health assessment and graded early warning method based on multimodal fusion, which is applied to an AI-based mental health assessment and graded early warning system based on multimodal fusion. The method includes the following steps:

[0047] The self-assessment results of the participants’ mental health were obtained using the PHQ-9 / GAD-7 scale.

[0048] During the self-assessment of the participants' mental health, facial expression data and physiological data were collected.

[0049] The psychological health level of the test subjects is determined based on the self-assessment results of psychological health, emotion recognition results, and physiological data, according to the pre-established multimodal psychological health assessment model.

[0050] The system displays the mental health level results and health guidance to the tested individuals, and provides tiered early warnings to healthcare professionals based on these mental health level results.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] This invention adds objective data such as physiological and facial expression data to the scale data. These are physiological and behavioral data indicators that are difficult to conceal in human emotional responses. This effectively reduces the subjective concealment by the subjects and forms a multimodal data that combines subjective and objective data for mental health assessment, thereby improving the accuracy of the assessment. Attached Figure Description

[0053] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0054] Figure 1 A block diagram of an AI-based mental health assessment and hierarchical early warning system based on multimodal fusion provided in an embodiment of the present invention;

[0055] Figure 2 A flowchart of an AI-based mental health assessment and hierarchical early warning method based on multimodal fusion provided in an embodiment of the present invention;

[0056] Figure 3 A diagram of the physical device of the AI-based mental health assessment and hierarchical early warning system based on multimodal fusion provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] like Figure 1 and Figure 3 As shown, this invention provides an AI-based mental health assessment and hierarchical early warning system based on multimodal fusion, comprising the following steps:

[0059] The assessment terminal module is used to allow test subjects to conduct self-assessment of mental health using the PHQ-9 / GAD-7 scale, and to determine the test subject's mental health level based on the self-assessment results, emotion recognition results, and physiological data according to the pre-established multimodal mental health assessment model. It also displays the mental health level results and health guidance to the test subjects, and provides graded warnings to medical staff based on the mental health level results.

[0060] The camera module is used to collect facial expression data of the test subjects during the self-assessment of their mental health, and to classify the emotions of the test subjects based on the facial expression data according to the pre-established emotion recognition model to obtain the emotion recognition results of the test subjects.

[0061] Wearable device module, used to collect physiological data of test subjects during the psychological health self-assessment process;

[0062] The synchronization optimization module, located between the assessment terminal module and the camera module, is used to maintain the synchronization between the multimodal mental health assessment model and the emotion recognition model through optimization.

[0063] This invention involves having subjects (such as cancer patients) complete the PHQ-9 / GAD-7 scale for self-assessment of mental health. To avoid concealment and deception that could reduce the reliability of the scale data, objective data—physiological data and facial expression data—are recorded during the self-assessment. The scale data, physiological data, and facial expression data (emotional state) are combined into multimodal data, and a multimodal mental health assessment model is established on this multimodal data. This allows for the analysis of the subjects' mental health status based on both subjective and objective data, and the data is categorized into levels. Early warnings are issued when mental health problems are found to be moderate or severe.

[0064] The main function of this invention in collecting facial expression data is to identify the emotional state of the test subject from the facial expression data. First, an emotion recognition model is constructed by training a neural network through a large number of facial expression images with emotion labels. It can identify the corresponding emotion category (anger, disgust, fear, happiness, normal, sadness, surprise, etc.) on the facial expression images, thereby grasping the objective and true emotional state of the test subject and using it in multimodal mental health assessment.

[0065] In this invention, all facial expression images collected during the self-evaluation process are collected. However, the emotion recognition model established in this invention performs emotion recognition on a single facial expression image. If each facial expression image in the self-evaluation process is to be recognized one by one, the model recognition calculation needs to be repeated multiple times, and multiple emotion recognition results may be generated, waiting for human decision-making. Therefore, the emotion recognition process is redundant and the addition of human decision-making will cause uncertainty.

[0066] To this end, the present invention first performs cluster analysis on all facial expression images during the self-assessment process, dividing each facial expression image into groups based on similarity. The cluster (or group) with the most facial expression images is likely to represent the true emotion of the test subject, and the facial expression image at the center of the cluster is representative of that cluster. Therefore, using the facial expression image at the center of the cluster for emotion recognition can realize the identification of the true emotion of the test subject through a single calculation of the model, without the need to perform recognition calculations on each facial expression image, reducing computational redundancy, and the recognition result is unique and definite, without the need for human decision-making.

[0067] The accuracy of the emotion recognition model in this invention largely depends on the quality of the facial expression images. During the self-evaluation process, due to the movement of the test subject or changes in the test environment, the collected facial expression images are prone to insufficient lighting and occlusion defects. Therefore, before performing emotion recognition, the facial expression images can be data augmented to repair insufficient lighting and occlusion defects.

[0068] The camera module includes a camera and a data processor;

[0069] The camera is used to collect facial expression data of the test subjects during the self-assessment of their mental health;

[0070] The data processor is used to classify the emotions of the test subject based on the subject's facial expression data using a pre-established emotion recognition model to obtain the emotion recognition result of the test subject.

[0071] An emotion recognition model is a classification network that takes an expression image as input and outputs an emotion category. The structural expression of the emotion recognition model is as follows:

[0072] ;

[0073] In the formula, The emotion category output by the emotion recognition model. For emoticons, For classification networks;

[0074] Emotion categories include anger, disgust, fear, happiness, normal, sadness, and surprise.

[0075] The ResNet network addresses the training difficulties of traditional deep networks through a residual learning framework and skip connections (shortcuts). Unlike traditional networks that directly learn the input-to-output mapping, ResNet breaks down complex functions into multiple residual mappings and uses skip connections to bypass some network layers and directly pass information, reducing the training difficulty. It is widely used in computer vision tasks such as image classification and detection.

[0076] The classification networks commonly used in this invention can also be VGG (Visual Geometry Group Network, the entire network consists of alternating convolutional and pooling layers, followed by fully connected layers and a Softmax classifier. Stacking deeper networks can improve the model's feature extraction capabilities, and due to its powerful feature extraction capabilities, it is often used for image classification) or DenseNet (Dense Convolutional Network, which establishes dense connections between all preceding layers and subsequent layers, so that each layer receives feature maps from all previous layers as input, and achieves feature reuse through concatenation along the channel dimension. This design reduces the number of parameters and computational cost while enhancing gradient propagation efficiency, and is also widely used in computer vision tasks such as image classification).

[0077] Data processors utilize pre-established emotion recognition models to classify the emotions of test subjects based on facial expression data, including:

[0078] All facial expression images collected from the test subjects during the psychological health self-assessment process were clustered using the K-means algorithm to obtain the facial expression image clustering results.

[0079] In the clustering results of facial expression images, select the facial expression image located at the center point in the cluster with the most images, input it into the emotion recognition model, and obtain the emotion category of the tested object;

[0080] The emotion category of the tested object is used as the emotion recognition result.

[0081] In this invention, after obtaining the center point image, its quality can be evaluated. If the quality of the facial expression image at the center point is found to be low or the clarity is insufficient, the image with the highest quality (such as the highest clarity or the highest degree of frontal view) in that cluster can be selected for replacement.

[0082] Wearable device modules include devices such as smartwatches, wristbands, and chest straps;

[0083] The physiological data collected by the wearable device module includes heart rate (HR), blood oxygen (SpO2), respiratory rate (RR), and heart rate variability (HRV).

[0084] The multimodal mental health assessment model takes as input self-assessment results of mental health, emotion recognition results, and physiological data, and outputs a classifier for mental health levels.

[0085] The structural expression of the multimodal mental health assessment model is as follows:

[0086] ;

[0087] In the formula, The mental health level output by the multimodal mental health assessment model. For the self-assessment results of mental health, According to the emotion category, This refers to all physiological data collected from the subjects during the self-assessment process of mental health. For classifiers, such as support vector machines, decision trees, random forests, etc.

[0088] This invention first trains a neural network using a large number of self-assessment results of mental health with mental health level labels, emotion recognition results, and physiological data to obtain a multimodal mental health assessment model that integrates multimodal data to evaluate mental health levels. This model enables the direct and objective assessment of the mental health level of the tested subject based on multimodal data. The assessment is highly automated, requires no human intervention, and adopts a unified assessment process with a high degree of standardization.

[0089] The multimodal mental health assessment model established in this invention relies on the output of the emotion recognition model for its input. Therefore, it is necessary to maintain the synchronization of the two models as much as possible. To this end, a network model is first established in the reverse direction of the input-output of the multimodal mental health assessment model. This model can predict the emotion recognition result (emotion category) based on the mental health level output by the multimodal mental health assessment model. This emotion recognition result comes from the emotion recognition model. According to the principle of model synchronization, the emotion recognition result predicted based on the mental health level should be consistent with the emotion recognition result output by the emotion recognition model. In other words, the difference between the two should be minimized to maintain the synchronization of the two network models. At the same time, the consistency between the two also indicates that the mental health level and the emotion recognition result have a unique and accurate mapping relationship. This shows that the mental health level output by the multimodal mental health assessment model has high accuracy and can accurately restore the clear recognition result.

[0090] Therefore, this invention aims to minimize the difference between the emotion recognition result predicted based on the psychological health level and the emotion recognition result output by the emotion recognition model. When the synchronization between the two is poor, the multimodal psychological health assessment model and the emotion recognition model are retrained until they are restored to synchronization.

[0091] During the retraining process of the two models, priority was assigned to each model. Higher priority models receive more emphasis on training. Priority is quantified by the individual model's loss; higher loss indicates a greater deterioration in performance, necessitating retraining to improve performance before reverting to the synchronized state. Therefore, in this invention, priority is aligned with the model's loss; the greater the loss, the higher the priority, and this priority is applied to the joint loss function of the two models during retraining. The higher the weight of each loss term, the more emphasis is placed on minimizing the loss term of that model during retraining, thus prioritizing the improvement of the model's performance. During the performance improvement of the two models, the optimization objective is also used for supervision until the performance of the two models is optimized to a synchronized state. In this way, the high performance of the two models is maintained while maintaining the synchronicity of the two models.

[0092] Methods for maintaining synchronization between the multimodal mental health assessment model and the emotion recognition model using the synchronization optimization module include:

[0093] Based on the input and output of the multimodal mental health assessment model, a convolutional neural network is established, taking the output of the multimodal mental health assessment model as input and the emotion recognition result as output. The structural expression of the convolutional neural network is as follows:

[0094] ;

[0095] In the formula, The emotion recognition result output by the convolutional neural network. The mental health level output by the multimodal mental health assessment model is represented by CNN, which stands for Convolutional Neural Network.

[0096] by and The differences between them serve as a synchronization target between multimodal mental health assessment models and emotion recognition models. , The L2 norm can be used, or the cross-entropy expression can be replaced in the loss calculation, depending on the need;

[0097] When the goal is synchronization If the value exceeds the preset threshold, the current multimodal mental health assessment model and emotion recognition model will be retrained to a synchronized state with the synchronization target as the optimization objective.

[0098] Among them, the retraining priority of the current multimodal mental health assessment model and emotion recognition model is related to their model loss function, and the priority is as follows:

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] The loss function for retraining is: ;

[0104] In the formula, Prioritizing the training of multimodal mental health assessment models The training priority for emotion recognition models, The loss function for the multimodal mental health assessment model. Let the loss function of the emotion recognition model be , For the true value of the emotion recognition result, For the true value of mental health level, The loss function for retraining;

[0105] When the goal is synchronization If the threshold is less than or equal to the preset threshold, the current multimodal mental health assessment model and emotion recognition model will remain unchanged.

[0106] This invention is for synchronization targets If the value exceeds a preset threshold, then the synchronization objective is used as the optimization goal, and the current multimodal mental health assessment model and emotion recognition model are retrained to a synchronized state. In other words, utilizing... The current multimodal mental health assessment model and emotion recognition model are retrained to continuously optimize them towards minimization. Until If the threshold is less than or equal to a preset threshold, retraining terminates. Both L and L are in their optimal state during retraining. There is also a mutual constraint relationship between L and it, which avoids random loss caused by retraining, that is, reduces the possibility that the retrained model will fail to improve performance and instead cause performance to decline.

[0107] In this invention, the priority is adapted to the model's own loss; the greater the loss, the higher the priority is set. The softmax function is then used to... and Transforming it into a probability distribution with a sum of 1 forms... and ,when The higher the value, the worse the performance of the multimodal mental health assessment model becomes, requiring retraining to improve its performance while restoring it to a state as synchronized as possible with the emotion recognition model. because The increase in the amount of total loss will lead to an increase in the total loss. The increased weighting raises the training priority of the multimodal mental health assessment model, which precisely meets the requirement that the greater the loss, the higher the priority setting. Similarly, when The higher the value, the worse the performance of the emotion recognition model becomes, requiring retraining to improve its performance while restoring it to a state as synchronized as possible with the multimodal mental health assessment model. because The increase in the amount of total loss will lead to an increase in the total loss. The increased weighting raises the training priority of the emotion recognition model, which perfectly meets the requirement that the greater the loss, the higher the priority should be.

[0108] Other priority determination methods can be used as needed, such as Dynamic Weight Averaging and Dynamic Task Prioritization, to make the priority adaptive and maintain the synchronization between models.

[0109] In order to eliminate the defect of inconsistent dimensions in the calculation process, the present invention allows for the normalization of the calculated data as needed during practical use.

[0110] Mental health levels are categorized as healthy, mild, moderate, and severe.

[0111] The assessment terminal module sends early warning information to medical staff when the mental health level is moderate or severe.

[0112] The assessment terminal module records each psychological health level result of the test subject and analyzes the psychological health trend of the test subject based on the results of multiple consecutive psychological health levels.

[0113] If the mental health of the tested subjects shows a deteriorating trend, a warning message will be sent to the medical staff.

[0114] This invention can monitor the mental health status of the tested subjects and analyze health trends, enabling continuous management of patients' mental health data. It creates a file for each individual and analyzes changes in mental health for patients who have undergone multiple tests, thereby serving as a warning to medical staff and ultimately benefiting patients.

[0115] like Figure 2 As shown, this invention provides an AI-based mental health assessment and hierarchical early warning method based on multimodal fusion, applied to an AI-based mental health assessment and hierarchical early warning system based on multimodal fusion. The method includes the following steps:

[0116] The self-assessment results of the participants’ mental health were obtained using the PHQ-9 / GAD-7 scale.

[0117] During the self-assessment of the participants' mental health, facial expression data and physiological data were collected.

[0118] The psychological health level of the test subjects is determined based on the self-assessment results of psychological health, emotion recognition results, and physiological data, according to the pre-established multimodal psychological health assessment model.

[0119] The system displays the mental health level results and health guidance to the test subjects, and provides tiered early warnings to medical staff based on the mental health level results.

[0120] This invention adds objective data such as physiological and facial expression data to the scale data. These are physiological and behavioral data indicators that are difficult to conceal in human emotional responses. This effectively reduces the subjective concealment by the subjects and forms a multimodal data that combines subjective and objective data for mental health assessment, thereby improving the accuracy of the assessment.

[0121] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. An AI-based mental health assessment and hierarchical early warning system based on multimodal fusion, characterized in that the system... include: The assessment terminal module is used to allow test subjects to conduct self-assessment of mental health using the PHQ-9 / GAD-7 scale, and to determine the test subject's mental health level based on the self-assessment results, emotion recognition results, and physiological data according to a pre-established multimodal mental health assessment model. It also displays the mental health level results and health guidance to the test subjects, and provides graded warnings to medical staff based on the mental health level results. The camera module is used to collect facial expression data of the test subjects during the self-assessment of their mental health, and to classify the emotions of the test subjects based on the facial expression data according to the pre-established emotion recognition model to obtain the emotion recognition results of the test subjects. Wearable device module, used to collect physiological data of test subjects during the psychological health self-assessment process; The synchronization optimization module is located between the assessment terminal module and the camera module, and is used to maintain the synchronization between the multimodal mental health assessment model and the emotion recognition model through optimization. The synchronization optimization module maintains synchronization between the multimodal mental health assessment model and the emotion recognition model using the following methods: Based on the input and output of the multimodal mental health assessment model, a convolutional neural network is established, taking the output of the multimodal mental health assessment model as input and the emotion recognition result as output. The structural expression of the convolutional neural network is as follows: ； In the formula, The emotion recognition result output by the convolutional neural network. The mental health level output by the multimodal mental health assessment model is represented by CNN, which stands for Convolutional Neural Network. by and The differences between them serve as a synchronization target between multimodal mental health assessment models and emotion recognition models. , It is an L2 norm; When the goal is synchronization If the value exceeds the preset threshold, the current multimodal mental health assessment model and emotion recognition model will be retrained to a synchronized state, with synchronization as the optimization objective.

2. The AI-based mental health assessment and hierarchical early warning system based on multimodal fusion according to claim 1, characterized in that: The camera module includes a camera and a data processor; The camera is used to collect facial expression data of the test subjects during the test subjects' self-assessment of mental health; The data processor is used to classify the emotions of the test subject based on the expression data of the test subject using a pre-established emotion recognition model to obtain the emotion recognition result of the test subject. The emotion recognition model is a classification network that takes an facial expression image as input and outputs an emotion category. The structural expression of the emotion recognition model is as follows: ； In the formula, The emotion category output by the emotion recognition model. For emoticons, For classification networks; The emotion categories include anger, disgust, fear, happiness, normal, sadness, and surprise.

3. The AI-based mental health assessment and hierarchical early warning system based on multimodal fusion according to claim 2, characterized in that: Data processors utilize pre-established emotion recognition models to classify the emotions of test subjects based on facial expression data, including: All facial expression images collected from the test subjects during the psychological health self-assessment process were clustered using the K-means algorithm to obtain the facial expression image clustering results. In the clustering results of facial expression images, select the facial expression image located at the center point in the cluster with the most images, input it into the emotion recognition model, and obtain the emotion category of the tested object; The emotion category of the tested object is used as the emotion recognition result.

4. The AI-based mental health assessment and hierarchical early warning system based on multimodal fusion according to claim 3, characterized in that: The wearable device module includes a smartwatch, a wristband, and a chest strap; The physiological data collected by the wearable device module includes heart rate (HR), blood oxygen (SpO2), respiratory rate (RR), and heart rate variability (HRV).

5. The AI-based mental health assessment and hierarchical early warning system based on multimodal fusion according to claim 4, characterized in that: The multimodal mental health assessment model takes as input self-assessment results of mental health, emotion recognition results, and physiological data, and outputs a classifier of mental health level; The structural expression of the multimodal mental health assessment model is as follows: ； In the formula, The mental health level output by the multimodal mental health assessment model. For the self-assessment results of mental health, According to the emotion category, This refers to all physiological data collected from the subjects during the self-assessment process of mental health. For classifiers.

6. The AI-based mental health assessment and hierarchical early warning system based on multimodal fusion according to claim 5, characterized in that: The retraining priority of current multimodal mental health assessment models and emotion recognition models is associated with their model loss functions, and the priority is as follows: ； ； ； ； The loss function for retraining is: ; In the formula, Prioritizing the training of multimodal mental health assessment models The training priority for emotion recognition models, The loss function for the multimodal mental health assessment model. Let the loss function of the emotion recognition model be , For the true value of the emotion recognition result, For the true value of mental health level, The loss function for retraining; When the goal is synchronization If the threshold is less than or equal to the preset threshold, the current multimodal mental health assessment model and emotion recognition model will remain unchanged.

7. The AI-based mental health assessment and hierarchical early warning system based on multimodal fusion according to claim 6, characterized in that: The mental health levels are categorized as healthy, mild, moderate, and severe.

8. The AI-based mental health assessment and hierarchical early warning system based on multimodal fusion according to claim 7, characterized in that: The assessment terminal module sends warning information to medical staff when the mental health level is moderate or severe.

9. The AI-based mental health assessment and hierarchical early warning system based on multimodal fusion according to claim 8, characterized in that: The assessment terminal module records each psychological health level result of the tested subject and analyzes the psychological health trend of the tested subject based on multiple consecutive psychological health level results. If the mental health of the tested subjects shows a deteriorating trend, a warning message will be sent to the medical staff.

10. A multimodal fusion-based AI-based method for mental health assessment and hierarchical early warning, characterized in that, The method applied to the AI-based mental health assessment and hierarchical early warning system based on multimodal fusion as described in any one of claims 1-9 includes the following steps: The self-assessment results of the participants’ mental health were obtained using the PHQ-9 / GAD-7 scale. During the self-assessment of the participants' mental health, facial expression data and physiological data were collected. The psychological health level of the test subjects is determined based on the self-assessment results of psychological health, emotion recognition results, and physiological data, according to the pre-established multimodal psychological health assessment model. The system displays the mental health level results and health guidance to the tested individuals, and provides tiered early warnings to healthcare professionals based on these mental health level results.

Images