Psychological state pre-screening system and method based on multi-modal data

Abstract

The invention discloses a psychological state pre-screening system and method based on multi-modal data. The psychological state pre-screening system comprises a data acquisition module, a data preprocessing module, a psychological state analysis module, a psychological state evaluation and screening module and a feedback and early warning module. The system collects multi-modal data such as facial expressions, voices, texts, physiological signals and the like, and performs psychological state analysis by using a deep learning algorithm. The psychological state score is calculated through a multi-modal data fusion algorithm, and feedback or early warning is provided, so that the user or related mechanisms can perform further intervention. According to the method, the accuracy and the real-time performance of psychological state screening can be improved, and the problem of misjudgment of a single data source is avoided. The system can be integrated to a smart phone, a smart bracelet, a computer and other equipment, realizes low-cost and non-perceptual mental health monitoring, is suitable for personal health management, enterprise employee mental monitoring, psychological counseling auxiliary diagnosis and school psychological screening, and is beneficial to early discovery and intervention of mental health problems.

Description

Technical Field

[0001] The present invention relates to the technical field of mental health monitoring, and in particular to a mental state pre-screening system and method based on multimodal data. Background Art

[0002] With the development of society and the accelerated pace of life, mental health has become a major public health issue worldwide. The incidence of mental illnesses such as anxiety, depression, and stress-related disorders continues to rise. Long-term untreated mental health issues can lead to serious physical and mental health damage, such as sleep disorders, decreased attention span, social withdrawal, and even extreme behavior. A World Health Organization (WHO) report indicates that over 300 million people worldwide suffer from depression, and the number of those with anxiety disorders is even greater. However, due to the hidden nature of mental health issues and social stigma, many patients do not receive timely and effective diagnosis and intervention.

[0003] Limitations of traditional mental state assessment methods:

[0004] Currently, psychological assessment relies primarily on the following methods: Psychological questionnaires and self-rating scales: Common scales include the Self-Rating Depressive Symptoms Questionnaire (PHQ-9) and the General Anxiety Scale (GAD-7). While these methods are highly standardized, they present several challenges: Subjectivity: Users may conceal their true mental state due to social desirability bias. Reliance on self-reported responses: Many respondents, lacking awareness of mental health or unwilling to acknowledge mental health issues, ignore or perfunctorily answer, compromising screening accuracy. Difficulty monitoring dynamic changes: Mental status can fluctuate over short periods of time, and questionnaires struggle to track emotional changes in real time. Psychological interviews: A professional psychologist assesses psychological status through face-to-face interviews. This method offers high accuracy, but has the following limitations: Cost: Psychological counseling resources are scarce, diagnostic costs are high, and widespread adoption is limited. Time-consuming: Each interview can take tens of minutes to several hours, making it unsuitable for rapid screening and large-scale application. Physiological signal detection: Mental status is assessed using data such as electroencephalogram (EEG), heart rate (HR), heart rate variability (HRV), and skin conductance (EDA). While these methods can provide relatively objective emotional indicators, they are expensive and have high barriers to entry: Many methods require specialized physiological sensors or medical-grade equipment, making them difficult to implement in daily life. Limitations of a single data source: Relying solely on physiological signals is susceptible to external environmental influences (such as movement and temperature), reducing screening accuracy. The rise of multimodal data analysis: In recent years, advances in artificial intelligence (AI) and big data analytics have made multimodal data-based psychological state assessment methods a research hotspot. Multimodal data refers to the comprehensive analysis of multiple data sources (such as facial expressions, voice, text, and physiological signals) to improve the accuracy of psychological state detection. Facial expression analysis: Micro-movements of facial muscles can reflect an individual's emotional state. Deep learning algorithms (such as CNNs and LSTMs) can automatically identify facial expression features and map them to psychological state assessment models. Speech emotion analysis: Speech characteristics such as pitch, velocity, and rhythm can reflect emotional fluctuations. Speech emotion analysis technologies based on deep neural networks (such as Bi-LSTM and CNN-RNN) have achieved high recognition accuracy. Sentiment Analysis: Natural language processing (NLP) technology can extract emotional information from users' written or spoken expressions. For example, pre-trained models based on the Transformer architecture (such as BERT and GPT) perform well in sentiment classification tasks. Physiological Signal Analysis: Heart rate variability (HRV) and skin conductance (EDA) can be used to detect individual psychological stress levels. Combined with machine learning algorithms, this can further optimize psychological state prediction.

[0005] Deficiencies of existing technologies: Although the above-mentioned mental state assessment technologies each have their own advantages, the following problems still exist:

[0006] A single data source is not sufficient to accurately assess psychological state: relying solely on facial expression analysis may affect accuracy due to factors such as ambient lighting and facial occlusion.

[0007] Relying solely on speech analysis is easily affected by external factors such as background noise and recording quality.

[0008] Relying solely on text analysis may not capture non-verbal emotional information, such as voice tone or facial expressions.

[0009] Relying solely on physiological signals may affect data stability due to external factors such as exercise and diet.

[0010] Multimodal data fusion still faces challenges: different modal data formats are different (such as images, audio, text, time series data, etc.), and how to effectively align and fuse data is a technical difficulty.

[0011] The weights of various modal data in psychological state assessment vary due to individual differences, and how to dynamically adjust the weights to adapt to individual characteristics still needs to be optimized.

[0012] Lack of real-time and universal accessibility: Most existing mental health testing methods require active user participation (such as filling out questionnaires, conducting interviews, etc.), and lack real-time, imperceptible mental state monitoring methods.

[0013] Many mental state analysis systems rely on specialized hardware and are not suitable for large-scale applications in daily life.

[0014] Therefore, we urgently need a psychological state pre-screening system and method based on multimodal data to solve the above problems. Summary of the Invention

[0015] The purpose of the present invention is to provide a psychological state pre-screening system and method based on multimodal data to solve the problems raised by the above background technology.

[0016] To achieve the above objectives, the present invention provides the following technical solution: a psychological state pre-screening system based on multimodal data, the system comprising:

[0017] A data acquisition module is used to collect multimodal data of the user, including but not limited to facial expression data, voice data, text data, and physiological signal data;

[0018] The data preprocessing module is used to denoise, standardize, and extract features from the collected multimodal data, and convert it into structured data suitable for analysis;

[0019] The psychological state analysis module is used to evaluate psychological state based on multimodal data. This module includes:

[0020] Expression analysis unit, used to analyze the user's facial expression changes and identify emotional states based on deep learning algorithms;

[0021] Speech analysis unit, used to extract speech features and make emotion judgments;

[0022] A text analysis unit, which is used to perform sentiment analysis and psychological state assessment on the text input by the user through natural language processing technology;

[0023] A physiological signal analysis unit is used to analyze the user's heart rate, skin conductance and other physiological indicators using wearable devices or sensors to assess psychological stress levels;

[0024] The psychological state assessment and screening module is used to screen the user's psychological state based on a multimodal data fusion algorithm and generate a psychological state score;

[0025] The feedback and warning module is used to provide feedback to users based on the results of psychological state analysis and to issue warnings when psychological abnormalities are detected.

[0026] As a preferred technical solution of the present invention, the data acquisition module uses a camera, microphone, smart bracelet or other sensor devices to collect data, and transmits the data to the data preprocessing module wirelessly or wiredly.

[0027] As a preferred technical solution of the present invention, the data preprocessing module includes:

[0028] Perform illumination compensation, face alignment and denoising on video data;

[0029] Perform noise reduction, feature extraction, and emotion modeling on speech data;

[0030] Perform word segmentation and sentiment classification on text data, and analyze it in combination with the psychology vocabulary library;

[0031] Normalize the physiological data and extract time series features.

[0032] As a preferred technical solution of the present invention, the mental state analysis module adopts a deep learning algorithm, including a convolutional neural network, a long short-term memory network or a Transformer-based model, to perform fusion analysis on multimodal data.

[0033] As a preferred technical solution of the present invention, the psychological state assessment and screening module adopts a weighted fusion algorithm to comprehensively process the analysis results of facial expressions, voice, text and physiological signals to improve the accuracy of psychological state assessment.

[0034] As a preferred technical solution of the present invention, the feedback and early warning module includes:

[0035] User feedback unit, used to provide personalized mental health advice based on mental state scores, including relaxation training, breathing regulation, music therapy, etc.;

[0036] The abnormal warning unit is used to send a warning notification to the user, family members or relevant institutions when an abnormal psychological state is detected.

[0037] As a preferred technical solution of the present invention, the system has personalized learning capabilities and can dynamically adjust the psychological state screening model based on user historical data to improve the accuracy of screening.

[0038] The present invention also provides a psychological state pre-screening method based on multimodal data, the method comprising the following steps:

[0039] Step S1: Data collection, collecting multimodal data of the user, including facial expressions, voice, text and physiological signal data;

[0040] Step S2: Data preprocessing, denoising, standardization and feature extraction of the collected data;

[0041] Step S3: Psychological state analysis, using deep learning algorithms to analyze multimodal data and extract emotional features;

[0042] Step S4: Psychological status assessment and screening: calculating the psychological status score based on the weighted fusion algorithm and performing screening;

[0043] Step S5: Feedback and warning, providing mental health advice to users based on the screening results, and sending abnormal warnings when necessary.

[0044] As a preferred technical solution of the present invention, the mental state analysis in step S3 adopts the Attention mechanism to optimize the feature fusion of different data sources and improve the accuracy of mental state prediction.

[0045] As a preferred technical solution of the present invention, the feedback and warning in step S5 set different feedback levels according to the threshold of the psychological state score, including normal, slightly abnormal, and need attention, so as to take corresponding mental health management measures.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention provides a psychological state pre-screening system and method based on multimodal data. By integrating multiple data sources such as facial expressions, voice, text, and physiological signals, the system utilizes deep learning algorithms to intelligently analyze psychological states. Compared to traditional single-source assessment methods, the present invention provides a more comprehensive and objective psychological state assessment, improving the accuracy and real-time nature of screening. Furthermore, by employing multimodal data fusion technology to dynamically weight different data sources, the system makes psychological state predictions more accurate, adapts to individual differences, and avoids the misjudgment problem associated with a single data source.

[0048] This invention can be widely used in personal health management, corporate employee mental health monitoring, auxiliary diagnosis by mental health institutions, and psychological screening in education. The system has strong universal applicability and can be integrated into everyday devices such as smartphones, smart bracelets, and computers, enabling low-cost, non-perceptible mental state monitoring. Furthermore, the invention also provides real-time feedback and early warning capabilities, providing intervention recommendations or notifying relevant agencies when mental states are abnormal, facilitating early detection and intervention of mental health issues, and improving users' mental health management capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a system block diagram of a psychological state pre-screening system based on multimodal data proposed by the present invention. DETAILED DESCRIPTION

[0050] The following is a combination of the embodiments of the present invention Figure 1 The technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] Example 1: Implementation of a mental state pre-screening system based on multimodal data;

[0052] This embodiment provides a psychological state pre-screening system based on multimodal data. The system analyzes the user's psychological state based on multiple data sources such as facial expressions, voice, text, and physiological signals, and provides screening results and warnings.

[0053] Data acquisition module: Facial expression data acquisition: The system uses a camera to capture the user's facial expressions, and performs face detection, expression recognition, and key point tracking through OpenCV combined with deep learning (such as MTCNN+FaceNet).

[0054] Voice data collection: The microphone is used to record the user's voice data. The system uses MFCC and Chroma features to extract voice signals and uses CNN-LSTM for emotion classification.

[0055] Text data collection: Users can input text through the keyboard, and the system uses the BERT model to perform text sentiment analysis and extract emotional features.

[0056] Physiological signal data acquisition: Smart bracelets or sensors are used to obtain physiological signals such as heart rate, heart rate variability (HRV), and skin conductance (EDA).

[0057] Data preprocessing module: denoises facial data, compensates for illumination, and performs face alignment.

[0058] The speech data is subjected to background noise removal, speech feature extraction, and converted into Mel spectrogram format.

[0059] The text data is segmented, stop words are removed, and sentiment classification is performed using the Library of Psychological Words (LIWC).

[0060] The physiological signal data were standardized and smoothed using a time series analysis model.

[0061] Psychological state analysis module: Expression analysis: Use the ResNet-50+LSTM model to classify user expression sequences into emotions (happy, sad, angry, surprised, neutral, etc.).

[0062] Speech analysis: Use Bi-LSTM to identify speech emotions and enhance them based on Prosody features.

[0063] Text analysis: Combines Transformer and a psychology vocabulary library to perform sentiment classification on user input text and calculate sentiment scores.

[0064] Physiological signal analysis: Random Forest and Support Vector Machine (SVM) are used to predict psychological state based on HRV, EDA and other data.

[0065] Psychological state assessment and screening module: The Attention mechanism is used to fuse multimodal data, and a decision tree is used for classification, ultimately dividing the user's psychological state into three levels: "normal", "mildly abnormal", and "needing attention".

[0066] Feedback and early warning module: When the user's mental state is normal, the system will display the normal state and suggest maintaining a good attitude.

[0067] In cases of mild abnormalities, the system recommends music, meditation or deep breathing exercises to improve emotional state.

[0068] If the system detects serious mental abnormalities, it will push psychological counseling suggestions to the user, and the user can choose to contact an emergency contact or a mental health agency.

[0069] Example 2: Mental health management application for corporate employees;

[0070] This embodiment provides an enterprise employee mental health management system based on multimodal data. The system can be embedded in the enterprise office platform to regularly screen employees' mental states to improve their well-being.

[0071] Data collection method: Integrate psychological state monitoring functions into employees' office equipment (such as laptop cameras, headset microphones). When employees participate in online meetings, the system can automatically collect their facial expressions and voice emotion characteristics.

[0072] When employees communicate in an enterprise chat system, the system can analyze the tone of the text to determine their mental state.

[0073] Wearable devices (such as smart bracelets) can be used to monitor employees' physiological data (heart rate, sleep quality).

[0074] Mental State Analysis: Facial Expression Recognition: Emotion classification is performed on conference camera data, and mental state prediction is performed based on facial muscle changes.

[0075] Voice emotion analysis: Analyzes parameters such as speech rate, pitch, and jitter to identify emotional states during conference calls.

[0076] Text sentiment analysis: Use natural language processing (NLP) algorithms based on chat logs to assess psychological states.

[0077] Psychological state screening and feedback: If the system detects that an employee is under great stress, it will provide relaxation training or mental health course recommendations in the enterprise health management system.

[0078] If the system detects a persistent abnormal mental state, an anonymous data analysis report can be sent to HR or mental health counselors for further support.

[0079] Example 3: Auxiliary diagnosis system for mental health institutions;

[0080] This embodiment provides a mental health auxiliary diagnosis system based on multimodal data. The system is used in mental health institutions to assist psychological counselors in performing intelligent mental state analysis and auxiliary diagnosis.

[0081] Data collection: The camera, microphone and physiological signal acquisition equipment in the psychological counseling room are used to automatically collect the patient's facial expressions, voice and physiological data.

[0082] By combining psychological questionnaires (such as PHQ-9 and GAD-7) with text input, patients' self-reported information is recorded and text sentiment analysis is performed.

[0083] Mental state analysis: Use deep learning models to assess the patient's emotions based on facial expressions, voice, and text data, and conduct comprehensive analysis in combination with physiological data.

[0084] The system can automatically generate a psychological status report, provide the patient's emotional trend, and compare it with historical data to assist mental health experts in judging the patient's psychological condition.

[0085] Screening and feedback: For general mood swings, the system provides self-help improvement suggestions such as meditation and relaxation training.

[0086] For patients with high-risk psychological conditions such as depression and anxiety, the system will remind mental health experts to conduct further assessments and may recommend that patients make appointments for psychological counseling or treatment.

[0087] Example 4: Student mental health screening in the education field;

[0088] This embodiment provides a mental health screening system for student groups, which is suitable for school mental health management, so as to detect students' mental problems early and provide intervention measures.

[0089] Data collection: Integrate psychological state monitoring functions into the learning software or online classroom system used by students to collect their facial expressions, voice and text data.

[0090] Combine physiological signals, such as sleep data obtained by smart bracelets, to analyze students' mental state.

[0091] Psychological state analysis: When students participate in online courses, the system analyzes their voice emotions and facial expressions, and evaluates their emotional state based on the content of their homework text.

[0092] If the system detects that a student is depressed or stressed, it will provide mental health advice, such as learning adjustment methods and relaxation training.

[0093] Screening and feedback: If a student’s mental state is abnormal, the system can recommend further assessment by the school’s mental health teacher.

[0094] If a student's mental health condition is seriously abnormal, necessary intervention suggestions can be provided to parents or professional psychological institutions.

[0095] The aforementioned examples demonstrate various application scenarios for a multimodal data-based mental state pre-screening system, including personal health management, corporate mental health monitoring, assisted diagnosis by mental health institutions, and mental health screening in education. Through multimodal data collection, deep learning analysis, and intelligent screening, the system enables efficient and accurate mental state assessment, provides personalized feedback and early warnings, and enhances mental health management.

[0096] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A psychological state pre-screening system based on multimodal data, characterized in that: The system includes: A data acquisition module is used to collect multimodal data of the user, including but not limited to facial expression data, voice data, text data, and physiological signal data; The data preprocessing module is used to denoise, standardize, and extract features from the collected multimodal data, and convert it into structured data suitable for analysis; The psychological state analysis module is used to evaluate psychological state based on multimodal data. This module includes: Expression analysis unit, used to analyze the user's facial expression changes and identify emotional states based on deep learning algorithms; Speech analysis unit, used to extract speech features and make emotion judgments; A text analysis unit, which is used to perform sentiment analysis and psychological state assessment on the text input by the user through natural language processing technology; A physiological signal analysis unit is used to analyze the user's heart rate and skin conductance physiological indicators using wearable devices or sensors to assess psychological stress levels; The psychological state assessment and screening module is used to screen the user's psychological state based on a multimodal data fusion algorithm and generate a psychological state score; The feedback and warning module is used to provide feedback to users based on the results of psychological state analysis and to issue warnings when psychological abnormalities are detected.

2. The mental state pre-screening system based on multimodal data according to claim 1, characterized in that: The data acquisition module uses a camera, microphone, smart bracelet or other sensor devices to collect data, and transmits the data to the data preprocessing module via wireless or wired means.

3. The mental state pre-screening system based on multimodal data according to claim 1, characterized in that: The data preprocessing module includes: Perform illumination compensation, face alignment and denoising on video data; Perform noise reduction, feature extraction, and emotion modeling on speech data; Perform word segmentation and sentiment classification on text data, and analyze it in combination with the psychology vocabulary library; Normalize the physiological data and extract time series features.

4. The mental state pre-screening system based on multimodal data according to claim 1, characterized in that: The mental state analysis module uses deep learning algorithms, including convolutional neural networks, long short-term memory networks, or Transformer-based models, to perform fusion analysis on multimodal data.

5. The mental state pre-screening system based on multimodal data according to claim 1, characterized in that: The psychological state assessment and screening module adopts a weighted fusion algorithm to comprehensively process the analysis results of facial expressions, voice, text and physiological signals to improve the accuracy of psychological state assessment.

6. The mental state pre-screening system based on multimodal data according to claim 1, characterized in that: The feedback and early warning module includes: A user feedback unit, which provides personalized mental health advice based on mental state scores, including relaxation training, breathing regulation, and music therapy; The abnormal warning unit is used to send a warning notification to the user, family members or relevant institutions when an abnormal psychological state is detected.

7. The mental state pre-screening system based on multimodal data according to claim 1, characterized in that: The system has personalized learning capabilities and can dynamically adjust the psychological state screening model based on user historical data to improve the accuracy of screening.

8. A psychological state pre-screening method based on multimodal data, characterized in that: The method comprises the following steps: Step S1: Data collection, collecting multimodal data of the user, including facial expressions, voice, text and physiological signal data; Step S2: Data preprocessing, denoising, standardization and feature extraction of the collected data; Step S3: Psychological state analysis, using deep learning algorithms to analyze multimodal data and extract emotional features; Step S4: Psychological status assessment and screening: calculating the psychological status score based on the weighted fusion algorithm and performing screening; Step S5: Feedback and warning, providing mental health advice to users based on the screening results, and sending abnormal warnings when necessary.

9. The method for pre-screening mental states based on multimodal data according to claim 8, characterized in that: The mental state analysis in step S3 uses the Attention mechanism to optimize the feature fusion of different data sources and improve the accuracy of mental state prediction.

10. The method for pre-screening mental states based on multimodal data according to claim 8, characterized in that: The feedback and warning in step S5 are set at different feedback levels according to the threshold of the psychological state score, including normal, slightly abnormal, and needing attention, so that corresponding psychological health management measures can be taken.

Images