Multi-modal mental health assessment system based on artificial intelligence

Abstract

The invention discloses a multi-mode mental health assessment system based on artificial intelligence, and belongs to the technical field of mental health assessment. A multi-modal mental health assessment system based on artificial intelligence comprises a data acquisition module, a multi-modal fusion module and a health assessment module. The psychological health assessment method and device solve the problem that in the prior art, in the process of collecting user information, the collection dimension is single, so that the user information is not comprehensive, and then the accuracy of psychological health assessment is affected, and the accuracy of psychological health assessment is improved by integrating multi-source data such as facial expressions, voices, texts and videos. The method effectively improves the objectivity, accuracy and personalized intervention capability of evaluation, greatly improves the overall evaluation accuracy through the fusion of facial expression recognition, voice pressure signal detection, text emotion feature recognition and physiological parameter monitoring technologies, effectively reduces the single-modal error through multi-modal data complementation, enables the user information to be more comprehensive, and improves the user experience. And the accuracy of mental health assessment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mental health assessment, and in particular to a multimodal mental health assessment system based on artificial intelligence. Background Art

[0002] Mental health assessments have long relied on psychological scales (such as the SCL-90 and PHQ-9) or clinical interviews. These are highly subjective, have long update cycles, and are unable to promptly reflect the dynamic fluctuations of a user's mental state. In recent years, some research has attempted to improve assessment efficiency by introducing technological means, such as analyzing emotional tendencies through social media text or using wearable devices to monitor physiological indicators such as heart rate variability.

[0003] The Chinese patent with publication number CN120260938A discloses a big data-based mental health assessment system, which includes a data acquisition module, a data processing module, a dynamic modeling module, a risk assessment module, and an intervention generation module. The data acquisition module acquires the user's heterogeneous data sources in real time through a distributed interface and generates structured data; the data processing module receives the structured data and generates a multimodal feature vector; the dynamic modeling module receives the multimodal feature vector, constructs a user psychological state map through a graph neural network, and generates a psychological state evolution signal based on the topological structure of the map updated by a time sliding window; the risk assessment module receives the psychological state evolution signal and forms an early warning signal; the intervention generation module receives the early warning signal, generates personalized intervention instructions, and transmits them to the user terminal. The big data-based mental health assessment system of the present invention can solve the problems of computational redundancy, poor geometric adaptability, and low knowledge distillation efficiency of the traditional attention mechanism when processing circular areas.

[0004] When the above patent is actually used to assess the user's mental health, the dimensions collected in the process of collecting user information are relatively single, resulting in incomplete user information, which in turn affects the accuracy of the mental health assessment; therefore, it does not meet existing needs. For this reason, we propose a multimodal mental health assessment system based on artificial intelligence. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal mental health assessment system based on artificial intelligence. By integrating multi-source data such as facial expressions, voice, text, and video, the objectivity, accuracy, and personalized intervention capabilities of the assessment are effectively improved. Through the integration of facial expression recognition, voice stress signal detection, text emotion feature recognition, and physiological parameter monitoring technology, the overall assessment accuracy is greatly improved. The complementary nature of multimodal data effectively reduces the error of a single modality, making user information more comprehensive, thereby improving the accuracy of mental health assessment and solving the problems raised in the above-mentioned background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal mental health assessment system based on artificial intelligence, comprising: A data acquisition module is used to collect data based on multiple data sources to obtain multimodal data acquisition results, and integrate the multimodal data acquisition results to obtain a multimodal data set; Multimodal fusion module, used to extract facial feature data, voice emotion feature data, text feature data, physiological feature data, and behavioral feature data, and fuse the extracted features; The health assessment module is used to retrieve historical mental health assessment files, build a mental health assessment model based on multimodal data, and use the mental health assessment model to obtain the user's mental health assessment results.

[0007] Preferably, the data acquisition module includes: An expression capture unit is used to detect and track the user's face and body contours, capture the user's facial expressions, behavioral movements, and eye movements during psychological stress testing, and obtain relevant video sets; A voice capture unit is used to collect relevant video sets and language expressions of users when performing psychological stress testing; The physiological data acquisition unit is used to obtain the user's physiological indicator monitoring data and behavior patterns.

[0008] Preferably, the multimodal fusion module includes: A facial feature extraction unit, configured to extract facial feature data using a captured video set related to psychological stress detection; A speech feature extraction unit is used to extract speech emotion feature data and text feature data using the captured psychological stress detection related video set and language expression; A physiological feature extraction unit, configured to extract physiological feature data and behavioral feature data using the acquired physiological indicator monitoring data and behavioral patterns; The feature fusion unit is used to fuse physiological feature data, facial feature data, behavioral feature data, voice emotion feature data and text feature data.

[0009] Preferably, the feature fusion unit specifically includes: Directly concatenate physiological feature data, facial feature data, and behavioral feature data to obtain high-dimensional joint vector features; The attention mechanism is used to fuse speech emotion feature data and text feature data to achieve cross-modal conversion from speech to text and obtain text emotion vector features; Use a linear layer to compress the high-dimensional joint vector features to the same dimension as the text sentiment vector features. After compression, concatenate the high-dimensional joint vector features and the text sentiment vector features to obtain a multimodal feature fusion vector. The principal component analysis method is used to remove redundant features in the multimodal feature fusion vector to obtain the optimized multimodal feature fusion vector.

[0010] Preferably, the facial feature extraction unit includes: The video is divided into static image sequences at fixed intervals, each frame is used as an independent input, and the MT convolutional neural network algorithm is used to locate the face area in each frame image and generate the bounding box coordinates; By locating the key points of the facial features, affine transformation is performed to adjust the face to a standard angle, reducing posture and lighting interference; Each frame of facial image is rotated, horizontally flipped, grayscaled, and resized to improve model robustness; Use convolutional neural networks to extract local features from each frame of facial image, capture abstract features through multi-layer stacking, flatten the feature map into a one-dimensional vector, and map it to the classification result; The extracted facial features are input into the LSTM network respectively. The LSTM learns the dynamic changes of facial and posture features in the time dimension, captures their temporal evolution process, and obtains the expression change trajectory of facial features, which is the facial feature data.

[0011] Preferably, the method of extracting speech emotion feature data using the captured psychological stress detection-related video set and language expression situation specifically includes: Adaptive dynamic range control technology is used to enhance the high-frequency components of the voice signal, ensuring that high-frequency information is effectively retained; In the PLP feature extraction process, a multi-resolution filter bank is introduced, combined with a Mel filter bank and a high-frequency compensation filter bank to simultaneously capture the low-frequency and high-frequency features of the speech signal; The extracted PLP features are used for spectrum enhancement, and short-time Fourier transform and inverse filtering techniques are used to further restore and enhance high-frequency information; The enhanced PLP features are input into the global semantic encoder, which uses a multi-scale attention mechanism to perform weighted fusion of low-frequency and high-frequency features to generate an audio global semantic vector; Extract the intonation and pitch changes from the global semantic vector of the audio, and combine it with the Mel-frequency cepstral coefficients to analyze the user's emotional characteristics.

[0012] Preferably, the health assessment module includes: A model building unit is used to retrieve historical mental health assessment files and build a mental health assessment model based on multimodal data; The result output unit is used to input the optimized multimodal feature fusion vector into the mental health assessment model to obtain the user's mental health assessment result.

[0013] Preferably, the model building unit specifically includes: A training unit, used to supervise the initial convolutional neural network training based on the training data set and the validation data set; A construction unit is used to retrieve historical mental health assessment files and build a mental health information assessment layer based on the collected multimodal data; The verification unit is used to synchronize the test data set to the mental health information assessment layer for testing. When the test passes, a mental health assessment model is obtained.

[0014] Preferably, the verification unit includes: a classification unit, configured to classify the test data set based on the level of mental health to obtain multiple test sets, configure a weighted weight for each test set based on the level of mental health, and determine an evaluation accuracy rate of each test set at the information evaluation layer, as well as an overall accuracy rate of all test sets at the information evaluation layer; A first evaluation unit is configured to calculate a first evaluation value for the information evaluation layer based on the weighted weight, the evaluation accuracy, and the overall accuracy; a data processing unit, configured to randomly lose and create contradictory data in the multimodal data in the test data set according to a preset ratio to obtain a reference test data set; A second evaluation unit is configured to calculate a second evaluation value for the information evaluation layer based on the accuracy and precision of the test data set and the reference test data set; A judging unit is configured to judge whether both the first evaluation value and the second evaluation value meet preset requirements. If so, the mental health assessment model is obtained; otherwise, the information assessment layer is adjusted again.

[0015] Preferably, the result output unit specifically includes: Setting evaluation indicators based on multimodal feature fusion vectors and determining multiple evaluation dimensions based on the evaluation indicators; Based on the weighted fusion results, a multidimensional assessment of mental health is conducted according to multiple assessment dimensions to generate a multidimensional assessment result of mental health.

[0016] Preferably, the construction of a mental health assessment model specifically includes: Divide the acquired multimodal data related to mental health into training dataset, validation dataset, and test dataset; Construct an initial convolutional neural network and perform supervised training on the initial convolutional neural network based on the training dataset and the validation dataset to obtain trained text processing layer, image processing layer, speech processing layer, and physiological parameter processing layer; Retrieve historical mental health assessment files and build a mental health information assessment layer based on the collected multimodal data; Connecting the outputs of the text processing layer, the image processing layer, the speech processing layer, and the physiological parameter processing layer to the input of the mental health information assessment layer; The test data set is synchronized to the mental health information assessment layer for testing. When the test passes, a mental health assessment model is obtained.

[0017] Preferably, the building block comprises: A graph construction unit is used to obtain quantitative assessment results and non-quantitative assessment results from historical mental health assessment files, obtain structural features of the quantitative assessment results, structure the non-quantitative assessment results based on the structural features to obtain structured assessment results, and fuse the quantitative assessment results and the structured assessment results to construct an assessment knowledge graph based on the fusion results; A rule establishing unit, configured to determine habit features and corresponding psychological result features based on the evaluation knowledge graph of the current user, and establish an initial evaluation rule for the current user based on the habit features and the corresponding psychological result features; A reference determination unit, configured to extract reference question clusters and their corresponding reference evaluation results based on the evaluation knowledge graphs of other users; a matching unit, configured to obtain abnormal data based on the collected multimodal data, match the abnormal data with a reference question cluster, and select a reference question cluster having a matching degree greater than a preset matching degree as a target question cluster; A rule expansion unit, configured to expand the initial evaluation rules using the target problem cluster and its corresponding reference evaluation results to obtain comprehensive evaluation rules; a rule modification unit, configured to obtain a state display type based on the multimodal data, establish a standard evaluation index for the state display type, modify the evaluation index in the comprehensive evaluation rule based on the standard evaluation index to obtain an index modification result, and adjust the evaluation threshold in the comprehensive evaluation rule based on the index modification result to obtain a target evaluation rule; An establishing unit is used to establish a mental health information evaluation layer based on the evaluation indicators and evaluation thresholds of the target evaluation rules.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention effectively improves the objectivity, accuracy and personalized intervention capabilities of the assessment by integrating multi-source data such as facial expressions, voice, text, and video. Through the integration of facial expression recognition, voice stress signal detection, text emotion feature recognition and physiological parameter monitoring technology, the overall assessment accuracy is greatly improved. The complementary multimodal data effectively reduces the error of a single modality, making user information more comprehensive, thereby improving the accuracy of mental health assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the modules of the multimodal mental health assessment system based on artificial intelligence of the present invention; Figure 2 This is a flow chart of the artificial intelligence-based multimodal mental health assessment system of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] In order to solve the problem that the existing technology collects user information from a single dimension when assessing the user's mental health, resulting in incomplete user information and affecting the accuracy of mental health assessment, please refer to Figure 1-Figure 2 , this embodiment provides the following technical solutions: The AI-based multimodal mental health assessment system includes: The data acquisition module is used to collect data based on multiple data sources, obtain multimodal data collection results, and integrate the multimodal data collection results to obtain a multimodal data set. The multimodal data includes speech emotion recognition data, facial micro-expression capture data, text semantic analysis data, physiological indicator monitoring data, and behavioral pattern data; Multimodal fusion module, used to extract facial feature data, voice emotion feature data, text feature data, physiological feature data, and behavioral feature data, and fuse the extracted features; The health assessment module is used to retrieve historical mental health assessment files, build a mental health assessment model based on multimodal data, and use the mental health assessment model to obtain the user's mental health assessment results.

[0022] By integrating multi-source data such as facial expressions, voice, text, and video, the objectivity, accuracy, and personalized intervention capabilities of the assessment are effectively improved. Through the integration of facial expression recognition, voice stress signal detection, text emotion feature recognition, and physiological parameter monitoring technology, the overall assessment accuracy is greatly improved. Multimodal data complementarity effectively reduces single-modal errors, making user information more comprehensive, thereby improving the accuracy of mental health assessments. For example, facial expressions can assist in verifying emotional expressions in voice, and text feedback can correct deviations in visual behavior.

[0023] Data acquisition module, including: An expression capture unit is used to detect and track the user's face and body contours, capture the user's facial expressions, behavioral movements, and eye movements during psychological stress testing, and obtain relevant video sets; A voice capture unit is used to collect relevant video sets and language expressions of users when performing psychological stress testing; The physiological data acquisition unit is used to obtain the user's physiological indicator monitoring data and behavior patterns.

[0024] Multimodal fusion module, including: A facial feature extraction unit, configured to extract facial feature data using a captured video set related to psychological stress detection; A speech feature extraction unit is used to extract speech emotion feature data and text feature data using the captured psychological stress detection related video set and language expression; A physiological feature extraction unit, configured to extract physiological feature data and behavioral feature data using the acquired physiological indicator monitoring data and behavioral patterns; The feature fusion unit is used to fuse physiological feature data, facial feature data, behavioral feature data, voice emotion feature data and text feature data.

[0025] Feature fusion unit, specifically including: Directly concatenate physiological feature data, facial feature data, and behavioral feature data to obtain high-dimensional joint vector features; The attention mechanism is used to fuse speech emotion feature data and text feature data to achieve cross-modal conversion from speech to text and obtain text emotion vector features; Use a linear layer to compress the high-dimensional joint vector features to the same dimension as the text sentiment vector features. After compression, concatenate the high-dimensional joint vector features and the text sentiment vector features to obtain a multimodal feature fusion vector. The principal component analysis method is used to remove redundant features in the multimodal feature fusion vector to obtain the optimized multimodal feature fusion vector.

[0026] The attention mechanism is used to fuse speech emotion feature data and text feature data. By dynamically adjusting the weight distribution, the attention mechanism can dynamically focus on the timbre features of the speech modality and the semantic features of the text modality for different sentiment analysis tasks, effectively separate redundant information, and extract the most critical feature combination for classification decisions. It can reduce invalid information interference, accelerate model convergence and improve computational efficiency.

[0027] Facial feature extraction unit, including: The video is divided into static image sequences at fixed intervals, each frame is used as an independent input, and the MT convolutional neural network algorithm is used to locate the face area in each frame image and generate the bounding box coordinates; By locating the key points of the facial features, affine transformation is performed to adjust the face to a standard angle, reducing posture and lighting interference; Each face frame is rotated, horizontally flipped, grayscaled, and resized to improve model robustness. Normalization: Pixel values ​​are scaled to the [0, 1] range, or standardized to a distribution with a mean of 0 and a variance of 1, accelerating model convergence. Use convolutional neural networks to extract local features in each facial image, such as edges and textures, and capture abstract features such as facial contours through multi-layer stacking. Flatten the feature map into a one-dimensional vector and map it to classification results, such as facial identity and expression category. The extracted facial features are input into the LSTM network respectively. The LSTM learns the dynamic changes of facial and posture features in the time dimension, captures their temporal evolution process, and obtains the expression change trajectory of facial features, which is the facial feature data.

[0028] Convolutional neural networks are used to extract facial features. Convolutional neural networks are particularly good at automatically extracting high-level features from image data and can effectively extract facial features such as facial expressions and posture changes from consecutive frames. Convolutional neural networks can capture the spatial hierarchical structure of images through multiple convolutional layers, which is very useful for understanding facial details such as eyes, mouth, nose, etc., as well as the overall shape. It can then efficiently process new video data, thereby improving the efficiency and accuracy of facial feature extraction.

[0029] The captured psychological stress detection related video set and language expression are used to extract speech emotion feature data, including: Adaptive dynamic range control technology is used to enhance the high-frequency components of the voice signal, ensuring that high-frequency information is effectively retained; In the PLP feature extraction process, a multi-resolution filter bank is introduced, combined with a Mel filter bank and a high-frequency compensation filter bank to simultaneously capture the low-frequency and high-frequency features of the speech signal; The extracted PLP features are used for spectrum enhancement, and short-time Fourier transform and inverse filtering techniques are used to further restore and enhance high-frequency information; The enhanced PLP features are input into the global semantic encoder, which uses a multi-scale attention mechanism to perform weighted fusion of low-frequency and high-frequency features to generate an audio global semantic vector; Extract the intonation and pitch changes from the global semantic vector of the audio, and combine it with the Mel-frequency cepstral coefficients to analyze the user's emotional characteristics.

[0030] Health assessment modules, including: A model building unit is used to retrieve historical mental health assessment files and build a mental health assessment model based on multimodal data; The result output unit is used to input the optimized multimodal feature fusion vector into the mental health assessment model to obtain the user's mental health assessment result.

[0031] Model building unit, specifically including: A training unit, used to supervise the initial convolutional neural network training based on the training data set and the validation data set; A construction unit is used to retrieve historical mental health assessment files and build a mental health information assessment layer based on the collected multimodal data; The verification unit is used to synchronize the test data set to the mental health information assessment layer for testing. When the test passes, a mental health assessment model is obtained.

[0032] In one embodiment, the verification unit includes: a classification unit, configured to classify the test data set based on the level of mental health to obtain multiple test sets, configure a weighted weight for each test set based on the level of mental health, and determine an evaluation accuracy rate of each test set at the information evaluation layer, as well as an overall accuracy rate of all test sets at the information evaluation layer; A first evaluation unit is configured to calculate a first evaluation value for the information evaluation layer based on the weighted weight, the evaluation accuracy, and the overall accuracy; The calculation formula for the first evaluation value is as follows: in, represents the first evaluation value, n represents the number of groups in the test set, represents a constant, represents a natural constant, with a value of 2.72. Indicates the amount of data in the test dataset, represents the amount of data in the i-th test set, represents the weighted weight of the i-th test set, represents the evaluation accuracy of the i-th test set, Indicates the overall accuracy; a data processing unit, configured to randomly lose and create contradictory data in the multimodal data in the test data set according to a preset ratio to obtain a reference test data set; A second evaluation unit is configured to calculate a second evaluation value for the information evaluation layer based on the accuracy and precision of the test data set and the reference test data set; The calculation formula for the second evaluation value is as follows: in, represents the second evaluation value, represents the accuracy weight, represents the precision weight, represents the accuracy of the test dataset, represents the accuracy of the reference test dataset, represents the accuracy of the reference test dataset, represents the precision of the reference test dataset; A judging unit is configured to judge whether both the first evaluation value and the second evaluation value meet preset requirements. If so, the mental health assessment model is obtained; otherwise, the information assessment layer is adjusted again.

[0033] In this embodiment, the more serious the mental health level is, the greater the weight of the corresponding test set is.

[0034] In this embodiment, the sum of the accuracy weight and the precision weight is 1, which is set according to actual needs.

[0035] The beneficial effects of the above design scheme are as follows: the classification unit avoids the neglect of key groups in traditional one-size-fits-all assessments by grouping the test dataset by mental health level and assigning weights to each group. The calculation of the first evaluation value simultaneously incorporates the evaluation accuracy of each group, the overall accuracy, the data volume proportion, and the weighted weight, ensuring that the model's evaluation ability for the entire sample is guaranteed while focusing on strengthening the evaluation accuracy of high-priority groups, ensuring that the model is more reliable in core application scenarios. The data processing unit constructs a reference test dataset that is closer to actual application by randomly dropping multimodal data and creating contradictory data. The second evaluation value evaluates the stability of the model in the presence of incomplete data and conflicts by comparing the accuracy and precision of the original test set with the reference test set. This design avoids the problem of the model performing well only in ideal clean data but failing in real noisy environments, improves the model's anti-interference ability, and replaces vague subjective judgments with dual quantitative indicators of the first and second evaluation values, so that the test pass has a clear numerical basis, making the constructed mental health assessment model more practical, reliable, and scientific, and meeting the actual needs of multimodal mental health assessment.

[0036] The result output unit specifically includes: Setting evaluation indicators based on multimodal feature fusion vectors and determining multiple evaluation dimensions based on the evaluation indicators; Based on the weighted fusion results, a multidimensional assessment of mental health is conducted according to multiple assessment dimensions to generate a multidimensional assessment result of mental health.

[0037] Construct a mental health assessment model, specifically including: Divide the acquired multimodal data related to mental health into training data sets, validation data sets, and test data sets; speech emotion recognition data, facial micro-expression capture data, text semantic analysis data, physiological indicator monitoring data, and behavioral pattern data; Construct an initial convolutional neural network and perform supervised training on the initial convolutional neural network based on the training dataset and the validation dataset to obtain trained text processing layer, image processing layer, speech processing layer, and physiological parameter processing layer; The text processing layer is trained based on the text semantic analysis data in the training dataset. The image processing layer is trained based on the facial micro-expression capture data and behavioral pattern data in the training dataset. The speech processing layer is trained based on the speech emotion recognition data in the training dataset. The physiological parameter processing layer is trained based on the physiological indicator monitoring data in the training dataset. Retrieve historical mental health assessment files and build a mental health information assessment layer based on the collected multimodal data; Connecting the outputs of the text processing layer, the image processing layer, the speech processing layer, and the physiological parameter processing layer to the input of the mental health information assessment layer; The test data set is synchronized to the mental health information assessment layer for testing. When the test passes, a mental health assessment model is obtained.

[0038] In one embodiment, the building block comprises: A graph construction unit is used to obtain quantitative assessment results and non-quantitative assessment results from historical mental health assessment files, obtain structural features of the quantitative assessment results, structure the non-quantitative assessment results based on the structural features to obtain structured assessment results, and fuse the quantitative assessment results and the structured assessment results to construct an assessment knowledge graph based on the fusion results; A rule establishing unit, configured to determine habit features and corresponding psychological result features based on the evaluation knowledge graph of the current user, and establish an initial evaluation rule for the current user based on the habit features and the corresponding psychological result features; A reference determination unit, configured to extract reference question clusters and their corresponding reference evaluation results based on the evaluation knowledge graphs of other users; a matching unit, configured to obtain abnormal data based on the collected multimodal data, match the abnormal data with a reference question cluster, and select a reference question cluster having a matching degree greater than a preset matching degree as a target question cluster; A rule expansion unit, configured to expand the initial evaluation rules using the target problem cluster and its corresponding reference evaluation results to obtain comprehensive evaluation rules; a rule modification unit, configured to obtain a state display type based on the multimodal data, establish a standard evaluation index for the state display type, modify the evaluation index in the comprehensive evaluation rule based on the standard evaluation index to obtain an index modification result, and adjust the evaluation threshold in the comprehensive evaluation rule based on the index modification result to obtain a target evaluation rule; An establishing unit is used to establish a mental health information evaluation layer based on the evaluation indicators and evaluation thresholds of the target evaluation rules.

[0039] In this embodiment, the quantitative evaluation results are, for example, tabular data, and the non-quantitative evaluation results are, for example, semantic analysis results of psychological problems, such as consultation records and doctor's comments.

[0040] In this embodiment, the habit characteristics include, for example, falling asleep after 2 a.m. 5 days a week for 3 consecutive months, with the sleep time fluctuating by more than 1 hour, and frequently checking the phone before going to bed. The corresponding psychological outcome characteristics are that the anxiety scale score continues to be higher than the critical value by 15%.

[0041] In this embodiment, the problem clusters include, for example, a social anxiety cluster and a sleep anxiety cluster.

[0042] In this embodiment, the state display type is based on the user's current scenario, behavior pattern and historical data, and the typical psychological states, such as relaxation, stress, and social avoidance, are the contextual anchor points for evaluation.

[0043] In this embodiment, the standard evaluation indicators select core observation items from multiple dimensions such as physiology, face, voice, behavior, and text for each state to ensure the comprehensiveness of the evaluation.

[0044] In this embodiment, through the corresponding design of states, indicators and thresholds, the evaluation rules can be adapted to the situation, avoiding the use of a single standard to measure psychological characteristics under different states, and significantly improving the accuracy of the evaluation.

[0045] The beneficial effects of the above design scheme are: by unifying the quantitative evaluation results and non-quantitative evaluation results into a structured knowledge graph, using the graph structure to capture the complex causal relationship of psychological problems, providing a comprehensive and accurate data basis for further analysis, extracting user-specific habits and corresponding psychological results from the knowledge graph, forming a personalized evaluation baseline, and forming problem clusters through cluster analysis of other users' knowledge graphs, and extracting common evaluation patterns. For new users or rare symptoms, the evaluation rules of similar groups can be quickly migrated to avoid modeling from scratch, matching abnormal features in multimodal data with reference problem clusters, dynamically expanding evaluation rules, and enriching the evaluation system for current users. Through the corresponding design of states, indicators and thresholds, the evaluation rules can be contextually adapted, avoiding the use of a single standard to measure psychological characteristics under different states, and significantly improving the accuracy of the evaluation.

[0046] Working principle: When using the artificial intelligence-based multimodal mental health assessment system of the present invention, according to Figure 1 and Figure 2 , including the following steps: S1: Detect and track the user's facial and body contours, capture the user's facial expressions, behavioral movements, and eye movements during psychological stress testing, and obtain relevant video sets. Collect relevant video sets and language expressions of the user during psychological stress testing to obtain the user's physiological indicator monitoring data and behavioral patterns; S2: Extract speech emotion feature data and text feature data using the captured psychological stress detection-related video set and language expression, and extract physiological feature data and behavioral feature data using the acquired physiological indicator monitoring data and behavioral patterns; S3: Fuse physiological feature data, facial feature data, behavioral feature data, voice emotion feature data, and text feature data to obtain a multimodal feature fusion vector and optimize it; S4: Construct an initial convolutional neural network, perform supervised training on the initial convolutional neural network based on the training dataset and the validation dataset, retrieve historical mental health assessment files, and construct a mental health information assessment layer based on the collected multimodal data. Use the validation dataset for verification, and obtain a mental health assessment model after passing the verification. S5: Input the optimized multimodal feature fusion vector into the mental health assessment model, set evaluation indicators based on the multimodal feature fusion vector, determine multiple evaluation dimensions based on the evaluation indicators, perform multidimensional mental health assessment according to multiple evaluation dimensions based on the weighted fusion results, and generate multidimensional mental health assessment results.

[0047] In summary, the multimodal mental health assessment system based on artificial intelligence of the present invention effectively improves the objectivity, accuracy and personalized intervention ability of the assessment by integrating multi-source data such as facial expressions, voice, text, and video. Through the integration of facial expression recognition, voice pressure signal detection, text emotion feature recognition and physiological parameter monitoring technology, the overall assessment accuracy is greatly improved. The complementary multimodal data effectively reduces the error of a single modality, making the user information more comprehensive, thereby improving the accuracy of mental health assessment. For example, facial expressions can assist in verifying the emotional expression in voice, and text feedback can correct the deviation of visual behavior. The attention mechanism is used to fuse the voice emotion feature data and the text feature data. The attention mechanism can dynamically adjust the weight distribution to target For different sentiment analysis tasks, it dynamically focuses on the timbre features of the speech modality and the semantic features of the text modality, effectively separates redundant information, and extracts the most critical feature combinations for classification decisions, which can reduce the interference of invalid information, accelerate model convergence and improve computational efficiency. It uses convolutional neural networks to extract facial features. Convolutional neural networks are particularly good at automatically extracting high-level features from image data and can effectively extract facial features from continuous frames, such as facial expressions, posture changes, etc. Convolutional neural networks can capture the spatial hierarchical structure of images through multiple convolutional layers, which is very useful for understanding facial details such as eyes, mouth, nose, etc., as well as the overall shape, and can then efficiently process new video data, thereby improving the efficiency and accuracy of facial feature extraction.

[0048] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0049] While the embodiments of the present invention have been shown and described, it will be apparent to those skilled in the art that various changes, modifications, substitutions, and alterations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. An artificial intelligence-based multimodal mental health assessment system, characterized by: include: A data acquisition module is used to collect data based on multiple data sources to obtain multimodal data acquisition results, and integrate the multimodal data acquisition results to obtain a multimodal data set; Multimodal fusion module, used to extract facial feature data, voice emotion feature data, text feature data, physiological feature data, and behavioral feature data, and fuse the extracted features; The health assessment module is used to retrieve historical mental health assessment files and obtain the evaluation indicators of the target assessment rules from the historical mental health assessment files. It establishes a mental health information assessment layer based on the evaluation indicators and evaluation thresholds of the target assessment rules, builds a mental health assessment model based on multimodal data, and uses the mental health assessment model to obtain the user's mental health assessment results.

2. The multimodal mental health assessment system based on artificial intelligence according to claim 1 is characterized in that: The data acquisition module includes: An expression capture unit is used to detect and track the user's face and body contours, capture the user's facial expressions, behavioral movements, and eye movements during psychological stress testing, and obtain relevant video sets; A voice capture unit is used to collect relevant video sets and language expressions of users when performing psychological stress testing; The physiological data acquisition unit is used to obtain the user's physiological indicator monitoring data and behavior patterns.

3. The multimodal mental health assessment system based on artificial intelligence according to claim 1 is characterized in that: The multimodal fusion module includes: A facial feature extraction unit, configured to extract facial feature data using a captured video set related to psychological stress detection; A speech feature extraction unit is used to extract speech emotion feature data and text feature data using the captured psychological stress detection related video set and language expression; A physiological feature extraction unit, configured to extract physiological feature data and behavioral feature data using the acquired physiological indicator monitoring data and behavioral patterns; The feature fusion unit is used to fuse physiological feature data, facial feature data, behavioral feature data, voice emotion feature data and text feature data.

4. The multimodal mental health assessment system based on artificial intelligence according to claim 3 is characterized in that: The feature fusion unit specifically includes: Directly concatenate physiological feature data, facial feature data, and behavioral feature data to obtain high-dimensional joint vector features; The attention mechanism is used to fuse speech emotion feature data and text feature data to achieve cross-modal conversion from speech to text and obtain text emotion vector features; Use a linear layer to compress the high-dimensional joint vector features to the same dimension as the text sentiment vector features. After compression, concatenate the high-dimensional joint vector features and the text sentiment vector features to obtain a multimodal feature fusion vector. The principal component analysis method is used to remove redundant features in the multimodal feature fusion vector to obtain the optimized multimodal feature fusion vector.

5. The multimodal mental health assessment system based on artificial intelligence according to claim 3 is characterized in that: The facial feature extraction unit comprises: The video is divided into static image sequences at fixed intervals, each frame is used as an independent input, and the MT convolutional neural network algorithm is used to locate the face area in each frame image and generate the bounding box coordinates; By locating the key points of the facial features, affine transformation is performed to adjust the face to a standard angle, reducing posture and lighting interference; Each frame of facial image is rotated, horizontally flipped, grayscaled, and resized to improve model robustness; Use convolutional neural networks to extract local features from each frame of facial image, capture abstract features through multi-layer stacking, flatten the feature map into a one-dimensional vector, and map it to the classification result; The extracted facial features are input into the LSTM network respectively. The LSTM learns the dynamic changes of facial and posture features in the time dimension, captures their temporal evolution process, and obtains the expression change trajectory of facial features, which is the facial feature data.

6. The multimodal mental health assessment system based on artificial intelligence according to claim 3 is characterized in that: The method of extracting speech emotion feature data by using the captured psychological stress detection-related video set and language expression conditions specifically includes: Adaptive dynamic range control technology is used to enhance the high-frequency components of the voice signal, ensuring that high-frequency information is effectively retained; In the PLP feature extraction process, a multi-resolution filter bank is introduced, combined with a Mel filter bank and a high-frequency compensation filter bank to simultaneously capture the low-frequency and high-frequency features of the speech signal; The extracted PLP features are used for spectrum enhancement, and short-time Fourier transform and inverse filtering techniques are used to further restore and enhance high-frequency information; The enhanced PLP features are input into the global semantic encoder, which uses a multi-scale attention mechanism to perform weighted fusion of low-frequency and high-frequency features to generate an audio global semantic vector; Extract the intonation and pitch changes from the global semantic vector of the audio, and combine it with the Mel-frequency cepstral coefficients to analyze the user's emotional characteristics.

7. The multimodal mental health assessment system based on artificial intelligence according to claim 1, characterized in that: The health assessment module includes: The model building unit is used to retrieve historical mental health assessment files and build a mental health assessment model based on multimodal data, including: Divide the acquired multimodal data related to mental health into training dataset, validation dataset, and test dataset; Construct an initial convolutional neural network and perform supervised training on the initial convolutional neural network based on the training dataset and the validation dataset to obtain trained text processing layer, image processing layer, speech processing layer, and physiological parameter processing layer; Retrieve historical mental health assessment files and build a mental health information assessment layer based on the collected multimodal data; Connecting the outputs of the text processing layer, the image processing layer, the speech processing layer, and the physiological parameter processing layer to the input of the mental health information assessment layer; Synchronize the test data set to the mental health information assessment layer for testing. When the test passes, a mental health assessment model is obtained. A result output unit is used to input the optimized multimodal feature fusion vector into the mental health assessment model to obtain the user's mental health assessment result; The result output unit specifically includes: Setting evaluation indicators based on multimodal feature fusion vectors and determining multiple evaluation dimensions based on the evaluation indicators; Based on the weighted fusion results, a multidimensional assessment of mental health is conducted according to multiple assessment dimensions to generate a multidimensional assessment result of mental health.

8. The multimodal mental health assessment system based on artificial intelligence according to claim 7, characterized in that: The model building unit specifically includes: A training unit, used to supervise the initial convolutional neural network training based on the training data set and the validation data set; A construction unit is used to retrieve historical mental health assessment files and build a mental health information assessment layer based on the collected multimodal data; The verification unit is used to synchronize the test data set to the mental health information assessment layer for testing. When the test passes, a mental health assessment model is obtained.

9. The multimodal mental health assessment system based on artificial intelligence according to claim 8, characterized in that: The verification unit includes: a classification unit, configured to classify the test data set based on the level of mental health to obtain multiple test sets, configure a weighted weight for each test set based on the level of mental health, and determine an evaluation accuracy rate of each test set at the information evaluation layer, as well as an overall accuracy rate of all test sets at the information evaluation layer; A first evaluation unit is configured to calculate a first evaluation value for the information evaluation layer based on the weighted weight, the evaluation accuracy, and the overall accuracy; a data processing unit, configured to randomly lose and create contradictory data in the multimodal data in the test data set according to a preset ratio to obtain a reference test data set; A second evaluation unit is configured to calculate a second evaluation value for the information evaluation layer based on the accuracy and precision of the test data set and the reference test data set; A judging unit is configured to judge whether both the first evaluation value and the second evaluation value meet preset requirements. If so, the mental health assessment model is obtained; otherwise, the information assessment layer is adjusted again.

10. The multimodal mental health assessment system based on artificial intelligence according to claim 8, characterized in that: The building block comprises: A graph construction unit is used to obtain quantitative assessment results and non-quantitative assessment results from historical mental health assessment files, obtain structural features of the quantitative assessment results, structure the non-quantitative assessment results based on the structural features to obtain structured assessment results, and fuse the quantitative assessment results and the structured assessment results to construct an assessment knowledge graph based on the fusion results; A rule establishing unit, configured to determine habit features and corresponding psychological result features based on the evaluation knowledge graph of the current user, and establish an initial evaluation rule for the current user based on the habit features and the corresponding psychological result features; A reference determination unit, configured to extract reference question clusters and their corresponding reference evaluation results based on the evaluation knowledge graphs of other users; a matching unit, configured to obtain abnormal data based on the collected multimodal data, match the abnormal data with a reference question cluster, and select a reference question cluster having a matching degree greater than a preset matching degree as a target question cluster; A rule expansion unit, configured to expand the initial evaluation rules using the target problem cluster and its corresponding reference evaluation results to obtain comprehensive evaluation rules; a rule modification unit, configured to obtain a state display type based on the multimodal data, establish a standard evaluation index for the state display type, modify the evaluation index in the comprehensive evaluation rule based on the standard evaluation index to obtain an index modification result, and adjust the evaluation threshold in the comprehensive evaluation rule based on the index modification result to obtain a target evaluation rule; An establishing unit is used to establish a mental health information evaluation layer based on the evaluation indicators and evaluation thresholds of the target evaluation rules.

Images