Multi-modal mental health assessment system and method based on interface type emotional interaction

Abstract

The invention discloses a multi-mode mental health assessment system and method based on interface type emotional interaction, and relates to the technical field of medical care. The system comprises a data collection unit, an emotion recognition unit, a feature extraction unit, an evaluation unit, a feedback unit and a tracking unit. The method comprises the following steps: performing feature extraction on collected information through a feature extraction unit, constructing a relation between modals, fusing the relation and mutual influence among key parts, images, voices and texts in feature information, and paying attention to the relation of information in the modals and between the modals by a modal information interaction module; in the modal information interaction module, a multi-head dot product attention mechanism is used to accurately select features associated with another modal, and a feature fusion module is used to select parts containing emotion information in the interaction features for classification. The classification accuracy of the multi-modal data is higher than the classification accuracy of the single-modal data, so that the accuracy of mental health assessment is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of medical care technology, and in particular to a multimodal mental health assessment system and method based on interface-based emotional interaction. Background Art

[0002] Mental health assessment refers to the comprehensive, systematic and in-depth analysis of individual or group psychological phenomena using interviews, observations and tests under the joint guidance of biological, psychological, social and medical models.

[0003] However, existing technologies still have some shortcomings when assessing users’ mental health:

[0004] 1. When evaluating the mental health of users, the existing technologies collect user information in a relatively single dimension, which results in incomplete user information and affects the accuracy of mental health assessment;

[0005] 2. At the same time, most existing technologies implement mental health assessment based on single-modal data classification. Since single-modality cannot build the connection between user images, voice and text, it affects the accuracy of user mental health assessment;

[0006] In response to the above problems, the inventors proposed a multimodal mental health assessment system and method based on interface-based emotional interaction to solve the above problems. Summary of the invention

[0007] In order to solve the above problems, the purpose of the present invention is to provide a multimodal mental health assessment system and method based on interface-based emotional interaction.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions: a multimodal mental health assessment system based on interface-based emotional interaction, the mental health assessment system comprising a data collection unit, an emotion recognition unit, a feature extraction unit, an assessment unit, a feedback unit and a tracking unit;

[0009] The data collection unit includes an expression capture module, a voice recording module, a text input module and a questionnaire module;

[0010] The emotion recognition unit includes a natural language processing module, a text processing module, an expression recognition module, a physiological signal analysis module and a voice emotion analysis module;

[0011] The feature extraction unit includes a modal feature extraction module, a modal information interaction module, a feature fusion module and a sentiment classification module;

[0012] The evaluation unit includes a model evaluation module and a risk warning module;

[0013] The feedback unit includes a report generation module and a result suggestion module;

[0014] The tracking unit includes a plan customization module, a progress monitoring module and a regular reminder module.

[0015] Preferably, the expression capturing module captures the user's facial expression through a camera device;

[0016] The voice recording module records the user's voice information;

[0017] The text input module allows the user to input text information through a text input device;

[0018] The questionnaire module allows users to fill in questionnaire data information in the form of questionnaires.

[0019] Preferably, the natural language processing module is used to process voice information, analyze the features of voice information, and determine the emotional state of the user;

[0020] The text processing module analyzes the text content input by the user and makes an emotional judgment;

[0021] The expression recognition module captures the user's facial expression through the recognition device and recognizes the user's facial expression and facial emotion changes;

[0022] The physiological signal analysis module reads the user's physiological indicators through the conductivity analysis device to determine the user's emotional state and stress state;

[0023] The speech emotion analysis module analyzes the speech information features and text information features through a deep learning algorithm to determine the user's psychological state.

[0024] Preferably, the voice information features include volume, intonation and speaking speed.

[0025] Preferably, the modal feature extraction module extracts text features from text content, and extracts image features and voice features from voice information and facial expressions;

[0026] The modality information interaction module calculates the similarity of each image in the image sequence and each word in the speech and text sequence through a multi-head dot product attention mechanism, builds the connection between the modalities, and uses the connection to obtain important information in the modality;

[0027] The feature fusion module selects key parts of the features;

[0028] The sentiment classification module analyzes the key part information of the feature to complete the sentiment classification.

[0029] Preferably, the model evaluation module calculates, analyzes and evaluates the feature information extracted by the feature extraction unit through a deep learning model to obtain a mental health value;

[0030] The risk warning module uses a threshold setting mechanism to issue a warning reminder if the user's mental health value is lower than the minimum threshold.

[0031] Preferably, the report generation module generates chart data based on the mental health values, so that users can understand their own mental health conditions;

[0032] The result suggestion module generates mental health improvement suggestions based on the mental health value.

[0033] Preferably, the plan customization module generates a plan table according to the generated mental health improvement suggestions, so that the user can improve his mental health according to the plan;

[0034] The progress monitoring module monitors the user's progress in improving mental health in real time;

[0035] The regular reminder module performs regular reminders according to the user's mental health improvement progress.

[0036] Preferably, the periodic reminders include voice reminders, text message reminders and email reminders.

[0037] The present invention also provides a method for a multimodal mental health assessment system based on interface-based emotional interaction, comprising the following steps:

[0038] S1. Data Collection

[0039] The data collection unit captures the user's facial expressions, records the user's voice information, and the user inputs psychological text information, as well as conducts a psychological health questionnaire survey on the user, so as to obtain comprehensive psychological data information of the user;

[0040] S2. Data Identification

[0041] The emotion recognition unit analyzes the user's voice information and recognizes the input text information to determine the user's emotional state and make emotional judgments. It also recognizes the user's facial expressions and reads the user's physiological indicators and stress conditions.

[0042] S3. Data feature extraction

[0043] The feature extraction unit extracts features from speech, text and facial expression information to obtain speech features, text features and image features, calculates the similarity of each word in each image, speech and text sequence, builds the connection between the modes, and uses the connection to obtain important information in the mode, and then performs fusion analysis on the key parts of the features to complete the emotion classification;

[0044] S4. Calculate mental health values

[0045] The extracted feature information is calculated, analyzed and evaluated through the deep learning model of the evaluation unit to obtain the mental health value;

[0046] S5. Mental health display and improvement suggestions

[0047] Generate chart data through feedback units and mental health values, and generate mental health improvement suggestions for users to understand their own mental health status;

[0048] S6. Implementation of the Mental Health Improvement Plan

[0049] A schedule is generated for the generated mental health improvement suggestions through the tracking unit, and the user's execution is monitored and reminded to ensure that the user completes the schedule and ultimately achieves mental health improvement.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. In the present invention, the user's facial expressions, voice information and questionnaire text information are collected in many aspects through the data collection unit to comprehensively obtain the user's psychological data information, so that the psychological health status results are more accurate when the user's psychological health status is subsequently calculated and analyzed;

[0052] 2. In the present invention, the feature extraction unit is used to extract features from the collected information, build connections between modalities, and fuse key parts of the feature information, the connections and mutual influences between images, voices, and texts. The modal information interaction module focuses on the connections between information within and between modalities. In the modal information interaction module, a multi-head dot product attention mechanism is used to accurately select features associated with another modality, and a feature fusion module is used to select the parts of the interactive features that contain emotional information for classification. Compared with existing technologies, the accuracy of multimodal data classification is higher than that of unimodal data classification, thereby effectively improving the accuracy of mental health assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a schematic diagram of the composition of the mental health assessment system of the present invention.

[0055] Figure 2 It is a schematic diagram of the method flow of the mental health assessment system of the present invention.

[0056] In the figure: 1. Data collection unit; 2. Emotion recognition unit; 3. Feature extraction unit; 4. Evaluation unit; 5. Feedback unit; 6. Tracking unit; 11. Expression capture module; 12. Voice recording module; 13. Text input module; 14. Questionnaire module; 21. Natural language processing module; 22. Text processing module; 23. Expression recognition module; 24. Physiological signal analysis module; 25. Voice emotion analysis module; 31. Modal feature extraction module; 32. Modal information interaction module; 33. Feature fusion module; 34. Emotion classification module; 41. Model evaluation module; 42. Risk warning module; 51. Report generation module; 52. Result suggestion module; 61. Plan customization module; 62. Progress supervision module; 63. Regular reminder module. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Example: Figure 1-2 As shown, the present invention provides a multimodal mental health assessment system based on interface-based emotional interaction, the mental health assessment system comprising a data collection unit 1, an emotion recognition unit 2, a feature extraction unit 3, an assessment unit 4, a feedback unit 5 and a tracking unit 6;

[0059] The data collection unit 1 includes an expression capture module 11, a voice recording module 12, a text input module 13 and a questionnaire module 14;

[0060] The expression capture module 11 captures the user's facial expression through a camera device, the voice recording module 12 records the user's voice information, the text input module 13 allows the user to input text information through a text input device, and the questionnaire module 14 allows the user to fill in questionnaire data information in the form of a questionnaire;

[0061] The emotion recognition unit 2 includes a natural language processing module 21, a text processing module 22, an expression recognition module 23, a physiological signal analysis module 24 and a voice emotion analysis module 25;

[0062] The natural language processing module 21 is used to process voice information, analyze the voice information features, and judge the user's emotional state. The text processing module 22 analyzes the text content input by the user and makes emotional judgments. The expression recognition module 23 captures the user's facial expressions through the recognition device and recognizes the user's facial expressions and facial emotional changes. The physiological signal analysis module 24 reads the user's physiological indicators through the conductivity analysis device to judge the user's emotional state and stress state. The voice emotion analysis module 25 analyzes the voice information features and text information features through the deep learning algorithm to judge the user's psychological state. The voice information features include volume, tone and speaking speed;

[0063] The feature extraction unit 3 includes a modal feature extraction module 31, a modal information interaction module 32, a feature fusion module 33 and a sentiment classification module 34;

[0064] The modal feature extraction module 31 extracts text features from the text content, and extracts image features and voice features from the voice information and facial expressions. The modal information interaction module 32 calculates the similarity of each image in the image sequence, each word in the voice and text sequence through a multi-head dot product attention mechanism, builds the connection between the modalities, and uses the connection to obtain important information in the modalities. The feature fusion module 33 selects the key parts of the features, and the sentiment classification module 34 analyzes the key part information of the features to complete the sentiment classification.

[0065] The evaluation unit 4 includes a model evaluation module 41 and a risk warning module 42;

[0066] The model evaluation module 41 calculates, analyzes and evaluates the feature information extracted by the feature extraction unit 3 through a deep learning model to obtain a mental health value. The risk warning module 42 uses a threshold setting mechanism to issue a warning reminder if the user's mental health value is lower than the minimum value of the threshold;

[0067] The feedback unit 5 includes a report generation module 51 and a result suggestion module 52;

[0068] The report generation module 51 generates chart data based on the mental health values ​​for users to understand their own mental health conditions, and the result suggestion module 52 generates mental health improvement suggestions based on the mental health values;

[0069] The tracking unit 6 includes a plan customization module 61, a progress monitoring module 62 and a regular reminder module 63;

[0070] The plan customization module 61 generates a plan based on the generated mental health improvement suggestions, so that users can improve their mental health according to the plan. The progress monitoring module 62 monitors the progress of users' mental health improvement in real time. The regular reminder module 63 provides regular reminders based on the progress of users' mental health improvement. Regular reminders include voice reminders, SMS reminders and email reminders.

[0071] The present invention also provides a method for a multimodal mental health assessment system based on interface-based emotional interaction, comprising the following steps:

[0072] S1. Data Collection

[0073] The data collection unit 1 captures the user's facial expressions, records the user's voice information, and the user inputs psychological text information, and conducts a psychological health questionnaire survey on the user, so as to comprehensively obtain the user's psychological data information;

[0074] S2. Data Identification

[0075] The emotion recognition unit 2 analyzes the user's voice information and recognizes the input text information, determines the user's emotional state and makes emotional judgments, recognizes the user's facial expressions, and reads the user's physiological indicators and stress conditions;

[0076] S3. Data feature extraction

[0077] The feature extraction unit 3 extracts features from the speech, text and facial expression information to obtain speech features, text features and image features, calculates the similarity of each word in each image, speech and text sequence, builds the connection between the modes, and uses the connection to obtain important information in the mode, and then performs fusion analysis on the key part of the feature information to complete the emotion classification;

[0078] S4. Calculate mental health values

[0079] The extracted feature information is calculated, analyzed and evaluated by the deep learning model of the evaluation unit 4 to obtain the mental health value;

[0080] S5. Mental health display and improvement suggestions

[0081] Generate chart data through feedback unit 5 and mental health values, and generate mental health improvement suggestions for users to understand their own mental health status;

[0082] S6. Implementation of the Mental Health Improvement Plan

[0083] The tracking unit 6 generates a schedule for the generated mental health improvement suggestions, and monitors and reminds the user's execution to ensure that the user completes the schedule and ultimately completes the mental health improvement.

[0084] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal mental health assessment system based on interface-based emotional interaction, characterized by: The mental health assessment system comprises a data collection unit (1), an emotion recognition unit (2), a feature extraction unit (3), an assessment unit (4), a feedback unit (5) and a tracking unit (6); The data collection unit (1) comprises an expression capture module (11), a voice recording module (12), a text input module (13) and a questionnaire module (14); The emotion recognition unit (2) comprises a natural language processing module (21), a text processing module (22), an expression recognition module (23), a physiological signal analysis module (24) and a speech emotion analysis module (25); The feature extraction unit (3) comprises a modal feature extraction module (31), a modal information interaction module (32), a feature fusion module (33) and a sentiment classification module (34); The evaluation unit (4) comprises a model evaluation module (41) and a risk warning module (42); The feedback unit (5) comprises a report generation module (51) and a result suggestion module (52); The tracking unit (6) comprises a plan customization module (61), a progress monitoring module (62) and a regular reminder module (63).

2. The multimodal mental health assessment system based on interface-based emotional interaction as claimed in claim 1, characterized in that: The expression capturing module (11) captures the user's facial expression through a camera device; The voice recording module (12) records the user's voice information; The text input module (13) allows the user to input text information through a text input device; The questionnaire module (14) allows users to fill in questionnaire data information in the form of a questionnaire.

3. The multimodal mental health assessment system based on interface-based emotional interaction as claimed in claim 1, characterized in that: The natural language processing module (21) is used to process voice information, analyze the characteristics of the voice information, and determine the emotional state of the user; The text processing module (22) analyzes the text content input by the user and makes an emotional judgment; The expression recognition module (23) captures the user's facial expression through a recognition device, and recognizes the user's facial expression and facial emotional changes; The physiological signal analysis module (24) reads the user's physiological indicators through the conductivity analysis device to determine the user's emotional state and stress state; The speech emotion analysis module (25) analyzes speech information features and text information features through a deep learning algorithm to determine the user's psychological condition.

4. The multimodal mental health assessment system based on interface-based emotional interaction as claimed in claim 3, characterized in that: The voice information features include volume, intonation and speaking speed.

5. The multimodal mental health assessment system based on interface-based emotional interaction as claimed in claim 3, characterized in that: The modal feature extraction module (31) extracts text features from the text content, and extracts image features and voice features from the voice information and facial expressions; The modality information interaction module (32) calculates the similarity of each image in the image sequence and each word in the speech and text sequence through a multi-head dot product attention mechanism, builds a connection between the modalities, and uses the connection to obtain important information in the modality; The feature fusion module (33) selects key parts of the features; The sentiment classification module (34) analyzes the key part information of the feature to complete the sentiment classification.

6. The multimodal mental health assessment system based on interface-based emotional interaction as claimed in claim 5, characterized in that: The model evaluation module (41) calculates, analyzes and evaluates the feature information extracted by the feature extraction unit (3) through a deep learning model to obtain a mental health value; The risk warning module (42) uses a threshold setting mechanism to issue a warning reminder if the user's mental health value is lower than the minimum value of the threshold.

7. The multimodal mental health assessment system based on interface-based emotional interaction as claimed in claim 6, characterized in that: The report generation module (51) generates chart data based on the mental health value, so that the user can understand his or her own mental health status; The result suggestion module (52) generates mental health improvement suggestions based on the mental health value.

8. The multimodal mental health assessment system based on interface-based emotional interaction as claimed in claim 7, characterized in that: The plan customization module (61) generates a plan table according to the generated mental health improvement suggestions, so that the user can improve his mental health according to the plan; The progress monitoring module (62) monitors the user's progress in improving mental health in real time; The regular reminder module (63) performs regular reminders according to the user's mental health improvement progress.

9. The multimodal mental health assessment system based on interface-based emotional interaction as claimed in claim 8, characterized in that: The periodic reminders include voice reminders, text message reminders and email reminders.

10. A method for applying the multimodal mental health assessment system based on interface-based emotional interaction as described in any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Data Collection The data collection unit (1) captures the user's facial expressions, records the user's voice information, and allows the user to input psychological text information, as well as conducts a psychological health questionnaire survey on the user, thereby comprehensively obtaining the user's psychological data information; S2. Data Identification The emotion recognition unit (2) analyzes the user's voice information and recognizes the input text information, determines the user's emotional state and makes an emotional judgment, recognizes the user's facial expression, and reads the user's physiological indicators and stress status; S3. Data feature extraction The feature extraction unit (3) extracts features from the speech, text and facial expression information to obtain speech features, text features and image features, calculates the similarity of each word in each image, speech and text sequence, builds a connection between the modes, and uses the connection to obtain important information in the modes, and then performs a fusion analysis on the key part of the feature information to complete the emotion classification; S4. Calculate mental health values The extracted feature information is calculated, analyzed and evaluated by a deep learning model of an evaluation unit (4) to obtain a mental health value; S5. Mental health display and improvement suggestions Generate chart data through the feedback unit (5) and mental health values, and generate mental health improvement suggestions for users to understand their own mental health conditions; S6. Implementation of the Mental Health Improvement Plan A tracking unit (6) generates a schedule for the generated mental health improvement suggestions, and monitors and reminds the user of the execution status, so as to ensure that the user completes the schedule and ultimately achieves mental health improvement.

Images