Multi-mode mental health intelligent assessment and intervention system and method

Abstract

The invention discloses a multi-mode mental health intelligent assessment and intervention system and method. The system comprises a scene simulation module, an image acquisition module, an image feature extraction module, a voice acquisition module, a voice feature extraction module, a mental health recognition module, an emotion recognition module, a picture selection module, an emotion correction module, a video teaching module, an emotion guide module, an emotion matching module and an alternative thinking guide module. According to the multi-mode intelligent assessment and intervention system and method for mental health, assessment does not depend on scale questionnaires and professional psychologists any more on the basis of multi-mode mental health assessment, the application scene is widened, the accuracy of assessment is improved, meanwhile, psychological intervention is not dull or interesting due to the electronic cognitive behavior therapy, and the psychological health assessment and intervention system and method are suitable for being popularized and applied. The contradiction emotion of the student on psychological intervention is reduced, and the participation degree of the student is improved.

Description

Technical Field

[0001] The present invention specifically relates to a multimodal mental health intelligent assessment and intervention system and method. Background Art

[0002] The mental health of college students is a hot topic nowadays. College students with good mental health can better cope with many pressures in life, study and work, explore their own potential, achieve their goals, and make contributions to society to the best of their ability.

[0003] There may be many factors that affect mental health. Family environment, learning and working environment, and social environment are all protective factors for mental health. However, even though many colleges and universities have opened psychological counseling rooms, depression and anxiety caused by mental illness are still common among college students. The mental health of college students is a social problem that urgently needs a solution.

[0004] The most commonly used method for testing mental health is the scale questionnaire. Since the answers to the scale questionnaire can be falsified artificially, the authenticity of the test conclusions needs to be further verified by the professional judgment of a doctor. Summary of the invention

[0005] The present invention provides a multimodal mental health intelligent assessment and intervention system and method to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0006] A multimodal mental health intelligent assessment and intervention system, comprising:

[0007] The scene simulation module is used to simulate the preset interactive scenes;

[0008] An image acquisition module is used to collect facial video information of the user during scene interaction;

[0009] An image feature extraction module, used to extract facial image features from the facial video information collected by the image acquisition module;

[0010] The voice collection module is used to collect the user's voice information during the scene interaction process;

[0011] A speech feature extraction module, used to extract speech features from the speech information collected by the speech collection module;

[0012] A mental health recognition module, used to obtain a mental health recognition result based on the facial image features extracted by the image feature extraction module and the voice features extracted by the voice feature extraction module;

[0013] An emotion recognition module, used to identify the emotion category of each image frame in the facial video information collected by the image collection module;

[0014] An image selection module is used to select multiple image frames with negative emotions from all image frames for users to identify;

[0015] The emotion correction module is used to display the selected image to the user, receive the emotion category identified by the user, and display the correct emotion category corresponding to the image frame to the user after the user selects the emotion category of the image frame;

[0016] The video teaching module is used to provide users with relevant knowledge about the consequences of negative beliefs through video explanations;

[0017] The emotion guidance module is used to display a number of positive emotion descriptions and negative emotion descriptions to the user, receive the interruption operation made by the user when the negative emotion description is identified, and score the received interruption operation according to whether it is correct or timely;

[0018] The emotion matching module displays several negative emotions and corresponding questioning negative emotion thoughts to the user, receives the user's matching operation, and scores the received matching operation based on whether it is correct or not;

[0019] The alternative thinking guidance module is used to show several negative thinking patterns to the user, receive the alternative thinking patterns selected by the user for the negative thinking patterns, and score the received alternative thinking patterns based on whether they are correct or not.

[0020] Furthermore, the speech feature extraction module is used to extract Mel-frequency cepstral coefficients from the speech signal, and use a convolutional neural network to extract speech features from the Mel-frequency cepstral coefficients.

[0021] Furthermore, the image feature extraction module is used to decompose the facial video information into multiple image frames, convert each image frame into a grayscale image, and use a convolutional neural network to extract facial image features of the image.

[0022] Furthermore, the mental health recognition module includes a trained multimodal recognition model, which receives the facial image features extracted by the image feature extraction module and the voice features extracted by the voice feature extraction module, and inputs the mental health recognition results.

[0023] Furthermore, the multimodal mental health intelligent assessment and intervention system also includes:

[0024] The scenario selection module is used for managers to select and determine the interaction scenario.

[0025] Furthermore, in the process where the picture selection module selects a plurality of image frames with negative emotions from all image frames for user identification, the time interval between the selected plurality of image frames with negative emotions is greater than a preset value.

[0026] Furthermore, the multimodal mental health intelligent assessment and intervention system also includes:

[0027] The report output module is used to output the user's evaluation report.

[0028] A multimodal mental health intelligent assessment and intervention method, applied to the aforementioned multimodal mental health intelligent assessment and intervention system, comprising:

[0029] S1: simulating a preset interactive scene through the scene simulation module, collecting facial video information of the user during the scene interaction process through the image acquisition module, extracting facial image features from the facial video information collected by the image acquisition module through the image feature extraction module, collecting voice information of the user during the scene interaction process through the voice acquisition module, extracting voice features from the voice information collected by the voice acquisition module through the voice feature extraction module, and obtaining a mental health recognition result through the mental health recognition module based on the facial image features extracted by the image feature extraction module and the voice features extracted by the voice feature extraction module;

[0030] S2: identifying the emotion category of each image frame in the facial video information collected by the image collection module through the emotion recognition module, selecting a plurality of image frames with negative emotions from all the image frames for the user to identify through the picture selection module, displaying the selected images to the user through the emotion correction module, receiving the emotion category identified by the user, and displaying the correct emotion category corresponding to the image frame to the user after the user selects the emotion category of the image frame;

[0031] S3: Outputting relevant knowledge about the consequences of negative beliefs to the user in the form of video explanations through the video teaching module;

[0032] S4: displaying a number of positive emotion descriptions and negative emotion descriptions to the user through the emotion guidance module, receiving an interruption operation made by the user when a negative emotion description is identified, scoring the received interruption operation according to whether it is correct or timely, and completing this step when the score exceeds a preset value;

[0033] S5: showing a number of negative emotions and a number of corresponding questioning negative emotional thoughts to the user through the emotion matching module, receiving the user's pairing operation, scoring according to whether the received pairing operation is correct or not, and completing this step when the score exceeds a preset value;

[0034] S6: showing a number of negative thinking patterns to the user through the alternative thinking guidance module, receiving an alternative thinking pattern selected by the user for the negative thinking pattern, scoring the received alternative thinking pattern according to whether it is correct or not, and completing this step when the score exceeds a preset value;

[0035] S7: re-execute step S1, and output a new mental health identification result through the mental health identification module.

[0036] Furthermore, step S1 to step S7 are completed in stages at certain time intervals.

[0037] Further, step S1 to step S7 are repeated twice to complete three cycles of testing.

[0038] The benefit of the present invention lies in the multimodal mental health intelligent assessment and intervention system and method provided. Based on multimodal mental health assessment, the assessment no longer relies on scale questionnaires and professional psychologists, broadens the application scenarios, and improves the accuracy of the assessment. At the same time, the combination of electronic cognitive behavioral therapy and interactive games makes psychological intervention no longer boring, reduces students' resistance to psychological intervention, and improves students' participation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 It is a schematic diagram of the multimodal mental health intelligent assessment and intervention system of the present invention. DETAILED DESCRIPTION

[0041] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0042] like Figure 1 Shown is a multimodal mental health intelligent assessment and intervention system of the present application, comprising: a scene simulation module, an image acquisition module, an image feature extraction module, a voice acquisition module, a voice feature extraction module, a mental health recognition module, an emotion recognition module, a picture selection module, an emotion correction module, a video teaching module, an emotion guidance module, an emotion matching module and an alternative thinking guidance module.

[0043] Specifically, the scene simulation module is used to simulate the preset interactive scene. Specifically, the multimodal mental health intelligent assessment and intervention system also has a scene selection module, which is used for managers to select and determine the interactive scene. In this application, the user can be a college student who receives multimodal mental health intelligent assessment and intervention, and the manager is the operator of the system. The administrator member needs to select a scene that may make college students feel anxious, such as an interview scene. The scene simulation module will provide a scene generated by the Unreal Engine, and users need to answer questions raised by the characters in the scene within the specified time limit and interact.

[0044] The image acquisition module is used to collect the facial video information of the user during the scene interaction process. The voice acquisition module is used to collect the voice information of the user during the scene interaction process, and the voice information includes the voice intonation.

[0045] The image feature extraction module is used to extract facial image features from the facial video information collected by the image acquisition module. Specifically, the image feature extraction module is used to decompose the facial video information into multiple image frames, convert each image frame into a grayscale image, and use a convolutional neural network to extract facial image features of the image.

[0046] The speech feature extraction module is used to extract speech features from the speech information collected by the speech collection module. In an embodiment of the present application, the speech feature extraction module is used to extract Mel-frequency cepstral coefficients from the speech signal and use a convolutional neural network to extract speech features from the Mel-frequency cepstral coefficients.

[0047] The mental health recognition module is used to obtain a mental health recognition result based on the facial image features extracted by the image feature extraction module and the speech features extracted by the speech feature extraction module. In the present application, the mental health recognition module includes a trained multimodal recognition model. The multimodal recognition model receives the facial image features extracted by the image feature extraction module and the speech features extracted by the speech feature extraction module, and inputs the mental health recognition result.

[0048] The emotion recognition module is used to identify the emotion category of each image frame in the facial video information collected by the image acquisition module. The picture selection module is used to select multiple image frames with negative emotions (such as fear, anger, and disgust) from all image frames for users to identify. The emotion correction module is used to display the selected image to the user, receive the emotion category identified by the user, and display the correct emotion category corresponding to the image frame to the user after the user selects the emotion category of the image frame. The user needs to judge which of the seven basic emotions (happiness, sadness, fear, anger, disgust, surprise, and contempt) the image frame contains, and then compare it with the results provided by the module. Through guidance, the user recalls the emotional reasons for the expression and the negative beliefs generated during the evaluation process.

[0049] It is understandable that several consecutive image frames with negative emotions are relatively similar. To avoid such a situation, in an embodiment of the present application, when the picture selection module selects multiple image frames with negative emotions from all image frames for user identification, the time interval between the selected multiple image frames with negative emotions is greater than a preset value.

[0050] The video teaching module is used to output relevant knowledge about the consequences of negative beliefs to users through video explanations. The consequences of negative beliefs are explained to users through interactive videos. Users stand from the perspective of bystanders and learn to understand the consequences of negative beliefs through interactive simulation videos.

[0051] The emotion guidance module is used to display several positive and negative emotion descriptions to the user, receive the interruption operation made by the user when identifying the negative emotion description, and score according to whether the received interruption operation is correct or timely. Specifically, several positive and negative beliefs are randomly displayed in the form of text and pictures. The user needs to distinguish the negative belief in a short time and press the interrupt key in time. The module will receive the user's interruption operation. When the user interrupts untimely or incorrectly, the module will give a point deduction prompt, otherwise it will get extra points.

[0052] The emotion matching module displays several negative emotions and corresponding several questioning negative emotional thoughts to the user, receives the user's pairing operation, and scores according to whether the received pairing operation is correct or not. Specifically, in the present application, the emotion matching module interacts with the user in a game-like manner and receives the negative beliefs identified by the user. In the game scene, the user needs to click on the furniture and objects that may contain clues, find all the information about negative beliefs (such as the note in the drawer saying "I am not experienced enough to do it well"), and the corresponding evidence of questioning negative beliefs (such as "I got full marks in last week's report" written behind the picture frame). Such information and evidence are regarded as a pair. When the user finds a pair of matching information and evidence, the module will prompt "One negative belief has been cleared", and the number of negative beliefs cleared and their total number will be displayed in the upper left corner of the screen.

[0053] The alternative thinking guidance module is used to show several negative thinking modes to the user, receive the alternative thinking modes selected by the user for the negative thinking modes, and score according to whether the received alternative thinking modes are correct or not. In the embodiment of the present application, the alternative thinking guidance module interacts with the user in a game mode to guide the user to learn alternative thinking. Specifically, the game visualizes negative beliefs into different monsters, each of which represents a common negative thinking (such as "I am not good enough" or "No one likes me"). These monsters have different characteristics and attack methods, and users need to defeat them through positive thinking cards. Users can use positive thinking cards to attack monsters. There are specific alternative thinking on each card, and these positive thinking are expressed through concise and easy-to-understand text and images. The design of the card allows users to intuitively see how to replace negative thinking with positive and rational thinking when facing negative thinking. When the user clears the monster's health bar by using a positive thinking card, it is considered that the user has defeated the negative thinking. The module will record whether the user has selected the correct alternative thinking card in each round of battle. A higher accuracy rate means that the user's mastery of alternative thinking is improving.

[0054] In the implementation of the present application, the multimodal mental health intelligent assessment and intervention system further includes: a report output module. The report output module is used to output the user's assessment report. The system will display the assessment results in the form of a visual report, and use colors and logos to distinguish changes, so that users can see at a glance the areas for improvement and areas that require further efforts.

[0055] Optionally, the entire system can be ported to an embedded hardware device and an APP can be created in conjunction with the iOS or Android system, so that users can implement mental health assessment and intervention in their daily lives. Through fun and serious games, users' resistance to mental health assessment can be reduced, and the accuracy of assessment and the effectiveness of intervention can be improved.

[0056] The present application also discloses a multimodal mental health intelligent assessment and intervention method, which is applied to the aforementioned multimodal mental health intelligent assessment and intervention system, specifically comprising:

[0057] Day 1: Use the scene simulation module to simulate the preset interactive scene, use the image acquisition module to collect the user's facial video information during the scene interaction process, use the image feature extraction module to extract facial image features from the facial video information collected by the image acquisition module, use the voice acquisition module to collect the user's voice information during the scene interaction process, use the voice feature extraction module to extract voice features from the voice information collected by the voice acquisition module, and use the mental health recognition module to obtain the mental health recognition result based on the facial image features extracted by the image feature extraction module and the voice features extracted by the voice feature extraction module.

[0058] Day 2: The emotion recognition module is used to identify the emotion category of each image frame in the facial video information collected by the image acquisition module. The picture selection module selects multiple image frames with negative emotions from all image frames for user identification. The emotion correction module displays the selected images to the user, receives the emotion category identified by the user, and displays the correct emotion category corresponding to the image frame to the user after the user selects the emotion category of the image frame.

[0059] Day 3: Through the video teaching module, users are provided with knowledge about the consequences of negative beliefs through video explanations.

[0060] Day 4: Use the emotion guidance module to show users several positive and negative emotion descriptions, receive interruptions from users when they identify negative emotion descriptions, score the interruptions based on whether they are correct and timely, and complete the step when the score exceeds a preset value.

[0061] Day 5: Use the emotion matching module to display several negative emotions and corresponding questioning negative emotional thoughts to the user, receive the user's pairing operation, score the received pairing operation based on whether it is correct or not, and complete this step when the score exceeds the preset value.

[0062] Day 6: Use the alternative thinking guidance module to show users several negative ways of thinking, receive the alternative ways of thinking selected by users for the negative ways of thinking, score the received alternative ways of thinking based on whether they are correct or not, and complete this step when the score exceeds the preset value.

[0063] Day 7: Re-execute the assessment of the first day, conduct a second assessment, and output new mental health recognition results through the mental health recognition module. Specifically, users need to complete tasks in the same scene, and the system will conduct mental health assessments through facial expressions and voice intonation. The system will then present a comparison of the assessment reports for the first and seventh days. The system will display these comparison results in a visual report, and use colors and logos to distinguish changes, so that users can see at a glance where improvements are made and where further efforts are needed. At the same time, users will receive a loop completion reward badge within the system and unlock the second loop and new assessment scenes, and users will be ranked within the server based on the number of days they check in. In addition, when the user's overall mental health score improves, the user will receive a points reward based on the improved score. Points can be exchanged for mental health education resources (such as professional articles, relaxing audio) or virtual items (such as personalized avatars or theme backgrounds) within the system to enhance the interactive experience.

[0064] The aforementioned seven days is one evaluation cycle. In the embodiment of the present application, preferably, the aforementioned seven-day cycle is repeated twice, and three cycles of testing are completed, totaling 21 days. The user will get a specific evaluation report comparison at the end of the third cycle, which contains the evaluation results of each time. According to the 21-day effect in psychology, after three cycles, the user's mental health will be significantly improved.

[0065] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A multimodal mental health intelligent assessment and intervention system, characterized in that: Include: The scene simulation module is used to simulate the preset interactive scenes; An image acquisition module is used to collect facial video information of the user during scene interaction; An image feature extraction module, used to extract facial image features from the facial video information collected by the image acquisition module; The voice collection module is used to collect the user's voice information during the scene interaction process; A speech feature extraction module, used to extract speech features from the speech information collected by the speech collection module; A mental health recognition module, used to obtain a mental health recognition result based on the facial image features extracted by the image feature extraction module and the voice features extracted by the voice feature extraction module; An emotion recognition module, used to identify the emotion category of each image frame in the facial video information collected by the image collection module; An image selection module is used to select multiple image frames with negative emotions from all image frames for users to identify; The emotion correction module is used to display the selected image to the user, receive the emotion category identified by the user, and display the correct emotion category corresponding to the image frame to the user after the user selects the emotion category of the image frame; The video teaching module is used to provide users with relevant knowledge about the consequences of negative beliefs through video explanations; The emotion guidance module is used to display a number of positive emotion descriptions and negative emotion descriptions to the user, receive the interruption operation made by the user when the negative emotion description is identified, and score the received interruption operation according to whether it is correct or timely; The emotion matching module displays several negative emotions and corresponding questioning negative emotion thoughts to the user, receives the user's matching operation, and scores the received matching operation based on whether it is correct or not; The alternative thinking guidance module is used to show several negative thinking patterns to the user, receive the alternative thinking patterns selected by the user for the negative thinking patterns, and score the received alternative thinking patterns based on whether they are correct or not.

2. The multimodal mental health intelligent assessment and intervention system according to claim 1, characterized in that: The speech feature extraction module is used to extract Mel-frequency cepstral coefficients from speech signals, and use a convolutional neural network to extract speech features from the Mel-frequency cepstral coefficients.

3. The multimodal mental health intelligent assessment and intervention system according to claim 1, characterized in that: The image feature extraction module is used to decompose facial video information into multiple image frames, convert each image frame into a grayscale image, and use a convolutional neural network to extract facial image features of the image.

4. The multimodal mental health intelligent assessment and intervention system according to claim 1, characterized in that: The mental health recognition module includes a trained multimodal recognition model, which receives the facial image features extracted by the image feature extraction module and the voice features extracted by the voice feature extraction module, and inputs the mental health recognition results.

5. The multimodal mental health intelligent assessment and intervention system according to claim 1, characterized in that: The multimodal mental health intelligent assessment and intervention system further comprises: The scenario selection module is used for managers to select and determine the interaction scenario.

6. The multimodal mental health intelligent assessment and intervention system according to claim 1, characterized in that: In the process of the picture selection module selecting a plurality of image frames with negative emotions from all image frames for user identification, the time interval between the selected plurality of image frames with negative emotions is greater than a preset value.

7. The multimodal mental health intelligent assessment and intervention system according to claim 1, characterized in that: The multimodal mental health intelligent assessment and intervention system further comprises: The report output module is used to output the user's evaluation report.

8. A multimodal mental health intelligent assessment and intervention method, applied to the multimodal mental health intelligent assessment and intervention system according to any one of claims 1 to 7, characterized in that: Include: S1: simulating a preset interactive scene through the scene simulation module, collecting facial video information of the user during the scene interaction process through the image acquisition module, extracting facial image features from the facial video information collected by the image acquisition module through the image feature extraction module, collecting voice information of the user during the scene interaction process through the voice acquisition module, extracting voice features from the voice information collected by the voice acquisition module through the voice feature extraction module, and obtaining a mental health recognition result through the mental health recognition module based on the facial image features extracted by the image feature extraction module and the voice features extracted by the voice feature extraction module; S2: identifying the emotion category of each image frame in the facial video information collected by the image collection module through the emotion recognition module, selecting a plurality of image frames with negative emotions from all the image frames for the user to identify through the picture selection module, displaying the selected images to the user through the emotion correction module, receiving the emotion category identified by the user, and displaying the correct emotion category corresponding to the image frame to the user after the user selects the emotion category of the image frame; S3: Outputting relevant knowledge about the consequences of negative beliefs to the user in the form of video explanations through the video teaching module; S4: displaying a number of positive emotion descriptions and negative emotion descriptions to the user through the emotion guidance module, receiving an interruption operation made by the user when a negative emotion description is identified, scoring the received interruption operation according to whether it is correct or timely, and completing this step when the score exceeds a preset value; S5: showing a number of negative emotions and a number of corresponding questioning negative emotional thoughts to the user through the emotion matching module, receiving the user's pairing operation, scoring according to whether the received pairing operation is correct or not, and completing this step when the score exceeds a preset value; S6: showing a number of negative thinking patterns to the user through the alternative thinking guidance module, receiving an alternative thinking pattern selected by the user for the negative thinking pattern, scoring the received alternative thinking pattern according to whether it is correct or not, and completing this step when the score exceeds a preset value; S7: re-execute step S1, and output a new mental health identification result through the mental health identification module.

9. The multimodal mental health intelligent assessment and intervention method according to claim 8, characterized in that: Steps S1 to S7 are completed in stages at certain time intervals.

10. The multimodal mental health intelligent assessment and intervention method according to claim 8, characterized in that: Repeat steps S1 to S7 twice to complete three cycles of testing.

Images