Emotional state evaluation system and method based on multiple modes

Abstract

The invention discloses a multi-modal-based emotional state evaluation method, which comprises the following steps: S1, collecting multi-modal data of a user to be tested, the multi-modal data comprising voice data, facial micro-expression data and facial heart rate data; s2, constructing a multi-modal deep neural network, inputting the multi-modal data into the multi-modal deep neural network, and dynamically adjusting the weight of the modal data by the multi-modal deep neural network by using an attention mechanism so as to output and obtain fusion features; and S3, judging the emotional state of the to-be-detected user based on the fused features. Accordingly, the invention further discloses a multi-mode-based emotional state evaluation system, and the multi-mode-based emotional state evaluation system can be used for implementing the emotional state evaluation method, so that the physiological status of the patient is more accurately and comprehensively considered, the emotional state of the patient is evaluated, and the accuracy of the emotional state evaluation is improved. Therefore, professional psychologists are assisted in diagnosing depression.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion assessment and analysis, and in particular to a multimodal-based emotion state assessment system and method. Background Art

[0002] As we all know, depression is a common mental illness, especially major depression, which has a negative impact on both individuals and society. In order to help doctors quickly diagnose and identify depression in clinical practice, patients with depression have different depressive symptoms, which manifest as different behaviors.

[0003] In clinical diagnosis, more than 70% of patients with early depression are unwilling to consult a psychologist, which worsens their condition. Psychologists often conduct face-to-face interviews based on the commonly used Diagnostic and Statistical Manual of Mental Disorders standards, which define nine categories of depressive symptoms and describe different behaviors in daily life. Traditional diagnostic methods such as the Beck Depression Inventory (BDI) and the Hamilton Depression Rating Scale (HAM-D) basically rely on questionnaire assessments. Although this is the most effective method for diagnosing depression, this detection method is time-consuming and labor-intensive, and has the defect of being highly subjective, and lacks a comprehensive evaluation strategy.

[0004] To this end, in view of the problems and defects of existing depression assessment methods, there is a need for a multimodal emotion assessment and analysis system and method that comprehensively considers the patient's physiological condition to assist professional psychologists in diagnosing depression. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multimodal emotional state assessment system and method to more accurately and comprehensively consider the patient's physiological condition and assess the patient's emotional state, thereby assisting professional psychologists in diagnosing depression.

[0006] In order to solve the above technical problems, the technical solution of the present invention is: a method for assessing emotional state based on multimodality, which comprises the steps of:

[0007] S1: Collecting multimodal data of the user to be tested, wherein the multimodal data includes voice data, facial micro-expression data, and facial heart rate data;

[0008] S2: construct a multimodal deep neural network, input the multimodal data into the multimodal deep neural network, and the multimodal deep neural network dynamically adjusts the weight of the modal data using an attention mechanism to output a fusion feature;

[0009] S3: Determine the emotional state of the user to be tested based on the fusion features.

[0010] In the above technical solution of the present invention, the multimodal emotional state assessment method designed by the present invention introduces deep integration of multimodal data in emotional state analysis, thereby surpassing the single-modal emotional analysis method. For example, in many existing technologies, emotional state judgment is based only on single voice data or single expression data, which may ignore some key emotional clues.

[0011] It can be seen that the innovation of the emotional state assessment method designed by the present invention lies in its ability to integrate multimodal data. It introduces deep integration of multimodal data in the judgment and analysis of emotional state, surpassing the analysis method of a single modality. It combines multiple physiological and behavioral signals such as voice data, facial micro-expression data and facial heart rate data, and realizes data fusion and analysis through the attention mechanism of multimodal deep neural networks (MDNN).

[0012] Compared with the existing technology, the present invention combines multiple physiological and behavioral signals such as voice, facial micro-expressions and facial heart rate to provide a more comprehensive emotional state analysis. This multimodal data fusion greatly improves the accuracy and sophistication of emotional state assessment, and has good promotion prospects.

[0013] Furthermore, in the multimodal emotional state assessment method of the present invention, in step S2, specifically:

[0014] S21: construct a multimodal deep neural network, and pre-set initial weights for each modal data based on the statistical analysis results of historical sample data;

[0015] S22: inputting the multimodal data into the multimodal deep neural network, and when detecting that a feature change of a certain modal data reflects the emotional state more significantly, the multimodal deep neural network automatically increases the weight of the corresponding modal data using a trained weight adjustment model;

[0016] S23: The multimodal deep neural network outputs fusion features.

[0017] Furthermore, in the multimodal emotional state assessment method described in the present invention, in step S23, the weight adjustment model automatically increases the weight of the corresponding modality data based on the situational factor, the historical data factor and the immediate feedback factor.

[0018] Furthermore, in the multimodal emotional state assessment method described in the present invention, in step S2, the fusion features include: sound audio features, speech and text features, facial image features and facial physiological features.

[0019] Furthermore, in the multimodal emotional state assessment method of the present invention, in step S3, a comprehensive judgment of the emotional state is performed based on the fusion features, specifically:

[0020] Based on the sound audio features and speech and text features, the speech speed, sound intonation and content are analyzed to determine the emotional state of the user to be tested; detailed analysis is performed based on the facial image features to determine the instantaneous emotional state of the user to be tested; and the facial heart rate changes of the user to be tested are analyzed through the facial physiological features to determine the emotional state of the user to be tested.

[0021] Furthermore, the multimodal emotional state assessment method of the present invention further includes step S4:

[0022] Within a preset time period, S1-S3 are repeated to obtain a plurality of emotional states of the user to be tested within the preset time period, and to determine a change trend of the emotional state of the user to be tested.

[0023] Furthermore, in the multimodal emotional state assessment method described in the present invention, in step S4, based on the multiple emotional states within the preset time period, a long short-term memory network is used to construct an emotional state time series model of the user to be tested, and the changing trend of the emotional state of the user to be tested is judged based on the emotional state time series model.

[0024] Correspondingly, another object of the present invention is to disclose an emotional state assessment system based on multimodality, which can be used to implement the emotional state assessment method of the present invention.

[0025] In the present invention, the designed multimodal emotional state assessment system includes: a multimodal interaction module, a data receiving module and a multi-level analysis module, wherein the multimodal interaction module is communicatively connected to the data receiving module, and the data receiving module is communicatively connected to the multi-level analysis module;

[0026] Among them, the multimodal interaction module is used to collect multimodal data, which includes voice data, facial micro-expression data and facial heart rate data, and send it to the data receiving module; the data receiving module integrates the multimodal data and generates corresponding fusion features; the multi-level analysis module evaluates the emotional state based on the fusion features.

[0027] Furthermore, in the multimodal emotional state assessment system described in the present invention, the multimodal interaction module includes: an audio capture unit, an image capture unit and a heart rate monitoring unit, and the audio capture unit, the image capture unit and the heart rate monitoring unit respectively collect voice data, facial micro-expression data and facial heart rate data.

[0028] Furthermore, in the multimodal-based emotional state assessment system described in the present invention, the data receiving module uses a multimodal deep neural network with an attention mechanism to integrate the multimodal data and generate corresponding fusion features.

[0029] The beneficial effect of the present invention is that in the multimodal-based emotion state assessment method designed by the present invention, the emotion state assessment method combines multiple physiological and behavioral signals such as voice, facial micro-expressions and facial heart rate, and realizes data fusion and analysis through a multimodal deep neural network combined with an attention mechanism. This multimodal data fusion greatly improves the accuracy and sophistication of emotion state assessment, and has good promotion prospects and application value.

[0030] In some preferred embodiments, the multimodal emotional state assessment method designed by the present invention not only analyzes the emotional state at a single moment, but also captures the dynamic process of emotional changes through continuous monitoring of multimodal data. Such a design enables it to detect subtle fluctuations in emotions during long-term monitoring, which is crucial for identifying the chronic development process of mental illnesses such as depression.

[0031] Accordingly, the multimodal-based emotional state assessment system designed in the present invention can be used to implement the emotional state assessment method described above in the present invention, which also has the above advantages and excellent effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The present invention is a flowchart of the steps of the multimodal emotional state assessment method in one embodiment. DETAILED DESCRIPTION

[0033] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0034] Research has found that despite existing technologies capable of emotion recognition, the diagnosis of depression, especially in the early stages, remains challenging. To this end, the present invention, through collaborative analysis of multimodal data, can capture subtle changes in the emotional expression of patients with depression, which may be overlooked in traditional single-modal analysis; this multimodal analysis not only helps to identify potential depressive symptoms, but also assists professional psychologists in more accurate labeling and diagnosis, thereby improving the sensitivity and specificity of diagnosis.

[0035] Based on this, please refer to Figure 1 As shown, the present invention designs a multimodal emotional state assessment method, which includes the steps of:

[0036] S1: Collecting multimodal data, where the multimodal data includes voice data, facial micro-expression data, and facial heart rate data;

[0037] S2: construct a multimodal deep neural network, input the multimodal data into the multimodal deep neural network, and the multimodal deep neural network dynamically adjusts the weight of the modal data using an attention mechanism to output a fusion feature;

[0038] S3: Perform a comprehensive judgment of the emotional state based on the fusion features.

[0039] From the above description, it can be seen that in the present invention, the inventor has optimized and designed a method for assessing emotional state based on multimodality. The innovation of this method for assessing emotional state lies in its ability to integrate multimodal data. It introduces deep integration of multimodal data in the judgment and analysis of emotional state, which goes beyond the analysis method of a single modality. It combines a variety of physiological and behavioral signals such as voice data, facial micro-expression data, and facial heart rate data, and realizes data fusion and analysis through multimodal deep neural networks (MDNN). It provides a more comprehensive emotional state analysis, which can greatly improve the accuracy and sophistication of emotional state assessment, and has good promotion prospects and application value.

[0040] When the multimodal-based emotional state assessment method is actually applied, priority can be given to interacting with the person being tested to collect multimodal data (voice data, facial micro-expression data, facial heart rate data) of the person being tested; among them, voice data can be extracted through a convolutional neural network (CNN), facial micro-expression data can be analyzed through a long short-term memory network (LSTM) to analyze dynamic changes in facial expressions, and facial heart rate data can be obtained through a specific physiological signal processing algorithm.

[0041] Then, the emotional state assessment method designed by the present invention can input these independently extracted multimodal data into a multimodal deep neural network, which uses an attention mechanism to dynamically adjust the weight of each modal data. The attention mechanism can intelligently allocate the importance of each modal data in emotional assessment according to its effectiveness in the current context, thereby achieving more accurate emotional state analysis.

[0042] Finally, the multimodal deep neural network can fuse the above multimodal data to obtain and output fusion features, so as to use the fusion features to make a comprehensive judgment of the emotional state; wherein, when making judgments, a multi-level classifier (for example, a random forest or a support vector machine) can be used to improve the accuracy and sophistication of the emotional assessment. This multi-level data fusion greatly improves the accuracy of emotional assessment, so that the multimodal emotional state assessment method designed by the present invention can capture subtle emotional changes that may be missed in traditional single-modal analysis.

[0043] Furthermore, in the multimodal emotional state assessment method of the present invention, in step S2, the following steps are specifically included:

[0044] S21: construct a multimodal deep neural network, and pre-set initial weights for each modal data based on the statistical analysis results of historical sample data;

[0045] S22: inputting the multimodal data into the multimodal deep neural network, and when detecting that a feature change of a certain modal data reflects the emotional state more significantly, the multimodal deep neural network automatically increases the weight of the corresponding modal data using a trained weight adjustment model;

[0046] S32: The multimodal deep neural network outputs fusion features.

[0047] Furthermore, in the multimodal emotional state assessment method described in the present invention, in step S23, the weight adjustment model automatically increases the weight of the corresponding modal data based on the situational factor, the historical data factor and the immediate feedback factor.

[0048] In the above technical solution of the present invention, an intelligent weight allocation mechanism is introduced into the emotional state assessment method designed by the present invention to dynamically adjust the influence of different modal data on the final diagnosis result according to their importance. The mechanism is implemented by using weighted multimodal deep neural networks (WMDNN) combined with attention mechanism.

[0049] In the above step S2, the emotional state assessment method designed by the present invention first sets initial weights for each modal data (such as voice, facial micro-expression, facial heart rate), and these initial weights can be set based on the statistical analysis results of historical data.

[0050] For example, by analyzing a large amount of sample data, the relative importance of each modality for emotion assessment in general can be determined; specifically, voice data may be given a 40% weight in the initial setting, while facial micro-expressions and facial heart rate may be given a 30% weight each.

[0051] In practical applications, the multimodal deep neural network can automatically increase the weight of the corresponding modal data using a trained weight adjustment model when it detects that the characteristic changes of a certain modal data reflect the emotional state more significantly. For example, in a situation where the facial expression of the user to be tested changes dramatically (such as sudden frowning, facial tension, etc.), the system will increase the weight of facial micro-expression data in real time, so that it occupies a larger proportion in the final emotional evaluation. Among them, the multimodal deep neural network can monitor the indicators of AI training, such as loss rate, accuracy, and then select different activation functions to adjust the weight of the modality; for example, during training, at each iteration, the loss rate of "voice data" gradually decreases, the accuracy gradually increases, and when the slope exceeds "micro-expression", it means that the voice data reflects the emotional state "significantly".

[0052] It should be noted that the intelligent weight allocation mechanism of the applied weight adjustment model relies on several key factors, which are learned by the system through a large amount of sample data during the training phase and adjusted in real time according to the actual situation during the inference phase:

[0053] Context Factor: The multimodal deep neural network automatically identifies which modalities are more important based on the current context. The setting of context factors is usually based on the performance of specific emotional characteristics. For example, when the voice of the user to be tested shows significant emotional fluctuations, the weight of the voice data will be increased;

[0054] Historical Data Factor: The multimodal deep neural network will refer to the historical data of the user to be tested to adjust the current weight. For example, if the voice data of the user to be tested is more sensitive to emotion assessment in the past records, the multimodal deep neural network will increase the weight of the voice data;

[0055] Real-time Feedback Factor: During the interaction with the user, the real-time feedback of the user (such as the real-time heart rate changes and instantaneous expression changes of the user to be tested) is analyzed in real time, and the weight of each mode is adjusted accordingly.

[0056] It can be seen that the goal of the weighted multimodal deep neural network combined with the attention mechanism in the emotional state assessment method of the present invention to form an intelligent weight allocation mechanism is to achieve flexible weight adjustment between different modal data to ensure that the user's true emotional state can be more accurately reflected based on the emotional expression in the current situation. This flexible weight allocation mechanism greatly enhances the accuracy and adaptability of the emotional state assessment method in complex emotional judgment, which is not available in traditional methods.

[0057] Furthermore, in the multimodal emotional state assessment method described in the present invention, in step S2, the fusion features include: sound audio features, speech and text features, facial image features and facial physiological features.

[0058] Furthermore, in the multimodal emotional state assessment method of the present invention, in step S3, a comprehensive judgment of the emotional state is performed based on the fusion features, specifically:

[0059] Based on the sound audio features and speech and text features, the speech speed, sound intonation and content are analyzed to determine the emotional state of the user to be tested; detailed analysis is performed based on the facial image features to determine the instantaneous emotional state of the user to be tested; and the facial heart rate changes of the user to be tested are analyzed through the facial physiological features to determine the emotional state of the user to be tested.

[0060] In the above technical solution of the present invention, the emotional state assessment method dynamically adjusts the weights of modal data by adding an attention mechanism. When the multimodal data comes from speech, facial micro-expressions and facial heart rate, the fusion features output by the multimodal deep neural network may include sound audio features, speech and text features, facial image features and facial physiological features.

[0061] At this time, in the above step S3, the emotional state assessment method can use the following three levels for specific analysis and judgment:

[0062] Based on the sound audio features and voice and text features in the fusion features, analyze the speech speed, voice intonation and content to determine the emotional state of the user to be tested;

[0063] Perform detailed analysis based on the facial image features in the fusion features to determine the instantaneous emotional state of the user to be tested;

[0064] The facial heart rate changes of the user to be tested are analyzed according to the facial physiological features in the fusion features to determine the emotional state of the user to be tested.

[0065] After the three levels of analysis and judgment have been made independently, the judgments on the emotional state of the user to be tested will be fused by weighted average. For example, one weight is given to the analysis results of speech speed, voice intonation and content, another weight is given to the analysis of facial image features, and another weight is given to facial heart rate changes. According to the relative importance of each modal data in the fused features, the judgment is integrated into an overall score, thereby outputting the final judgment result on the user's emotional state.

[0066] Furthermore, the multimodal emotional state assessment method described in the present invention also includes step S4: repeating the above steps S1-S3 within a preset time period to obtain multiple emotional states within the preset time period, and judging the changing trend of the emotional state of the user to be tested.

[0067] Furthermore, in the multimodal emotional state assessment method described in the present invention, in step S4, based on the multiple emotional states within the preset time period, a long short-term memory network is used to construct an emotional state time series model of the user to be tested, and the changing trend of the emotional state of the user to be tested is judged based on the emotional state time series model.

[0068] The multimodal emotional state assessment method described in the present invention does not only analyze the emotional state at a single moment, but also captures the dynamic process of emotional changes through continuous monitoring of multimodal data.

[0069] In the process of confirming depression, the emotional state assessment method designed by the present invention first obtains multiple emotional states of the user to be tested in the preset time period by continuously collecting multimodal data (such as voice, facial micro-expression, and facial heart rate) in a preset time period, and then uses a long short-term memory network (LSTM) to build a time series model of the user's emotional state, thereby processing and analyzing the time series of the emotional state to capture the user's emotional fluctuations in different time periods. For example, the voice data, facial micro-expression data, and facial heart rate data of the user to be tested are collected in a fixed time period (such as morning and evening) every day, and the emotional state of the user to be tested is analyzed.

[0070] In time series analysis, the system automatically detects the changing trend of emotional state. For example, if the user's emotional state gradually becomes depressed, speech speed slows down, facial expressions decrease, facial heart rate decreases, etc. during continuous monitoring for several weeks, a preliminary depression risk assessment will be given.

[0071] This continuous analysis not only helps identify early symptoms of depression, but also can evaluate the progression or relief of depression through dynamic monitoring. For example, during the treatment process, the effectiveness of the treatment can be evaluated by comparing the time series of emotional states before and after treatment, and data support can be provided to doctors to help adjust the treatment plan; traditional diagnostic methods often rely on assessments at fixed time points and ignore changes in emotions over time. The present invention can provide more detailed information on emotional fluctuations through continuous emotional state monitoring, thereby more accurately identifying the chronic development process of mental illnesses such as depression.

[0072] Correspondingly, the present invention also discloses a multimodal-based emotional state assessment system, which is specifically applied to the multimodal-based emotional state assessment method of the present invention, and comprises: a multimodal interaction module, a data receiving module and a multi-level analysis module, wherein the multimodal interaction module is communicatively connected to the data receiving module, and the data receiving module is communicatively connected to the multi-level analysis module;

[0073] Among them, the multimodal interaction module is used to collect multimodal data, which includes voice data, facial micro-expression data and facial heart rate data, and send it to the data receiving module; the data receiving module integrates the multimodal data and generates corresponding fusion features; the multi-level analysis module evaluates the emotional state based on the fusion features.

[0074] Furthermore, in the multimodal emotional state assessment system described in the present invention, the multimodal interaction module includes: an audio capture unit, an image capture unit and a heart rate monitoring unit, and the audio capture unit, the image capture unit and the heart rate monitoring unit respectively collect voice data, facial micro-expression data and facial heart rate data.

[0075] Furthermore, in the multimodal-based emotional state assessment system described in the present invention, the data receiving module uses a multimodal deep neural network that dynamically adjusts the weights of the modal data using an attention mechanism to integrate the multimodal data and generate corresponding fusion features.

[0076] Embodiment 1:

[0077] Please refer to Figure 1As shown, the first embodiment of the present invention is: a multimodal-based emotional state assessment system, which includes: a multimodal interaction module, a data receiving module and a multi-level analysis module, the multimodal interaction module is communicatively connected with the data receiving module, and the data receiving module is communicatively connected with the multi-level analysis module. In this embodiment, the multimodal interaction module specifically includes: an audio capture unit, an image capture unit and a heart rate monitoring unit, and the audio capture unit, the image capture unit and the heart rate monitoring unit respectively collect voice data, facial micro-expression data and facial heart rate data.

[0078] The emotional state assessment system designed by the present invention can interact with the person to be tested and collect multimodal data using the above-mentioned multimodal interaction module. The multimodal data specifically includes: voice data, facial micro-expression data and facial heart rate data.

[0079] Accordingly, in the emotional state assessment system designed by the present invention, the data receiving module can use a multimodal deep neural network with an attention mechanism to integrate the above-mentioned multimodal data, that is, input the multimodal data into the multimodal deep neural network, and the multimodal deep neural network uses the attention mechanism to dynamically adjust the weights of each modal data in the multimodal data and generate corresponding fusion features.

[0080] In practical application, the above data receiving module can be used to perform the following steps:

[0081] Construct a multimodal deep neural network and set initial weights for each modal data in advance based on the statistical analysis results of historical sample data;

[0082] The multimodal data is input into the multimodal deep neural network, and when it is detected that the characteristic change of a certain modal data reflects the emotional state more significantly, the multimodal deep neural network automatically increases the weight of the corresponding modal data using the trained weight adjustment model;

[0083] The multimodal deep neural network outputs the fused features.

[0084] It should be noted that, in actual implementation, the weight adjustment model can automatically increase the weight of the corresponding modal data based on the context factor, historical data factor and immediate feedback factor. In addition, in this embodiment, the fusion features output by the multimodal deep neural network can specifically include: sound audio features, voice and text features, facial image features and facial physiological features.

[0085] Accordingly, the data receiving module can output the fused features obtained by processing to the multi-level analysis module, so as to use the multi-level analysis module to receive the fused features. The multi-level analysis module can evaluate the emotional state according to the sound audio features, voice and text features, facial image features and facial physiological features in the fused features, and through the following levels:

[0086] Speech analysis: Based on the audio and speech characteristics of the user to be tested and the speech text characteristics, the speech speed, voice intonation and content are analyzed to determine the emotional state of the user to be tested.

[0087] Micro-expression analysis: Detailed analysis is performed based on the facial image features of the user to be tested to determine the user's instantaneous emotional state.

[0088] Physiological state analysis: Analyze the facial heart rate changes of the user to be tested through the facial physiological characteristics of the user to be tested to determine the emotional state of the user to be tested.

[0089] It can be seen that the innovation of the emotional state assessment system lies in its ability to integrate multimodal data, which can comprehensively analyze the multimodal indicators of the person to be tested (such as speaking speed, tone, micro-expressions and heart rate, etc.) to obtain a comprehensive assessment of the emotional state. Although the emotional state assessment system provides a preliminary judgment of the emotional state, the final labeling and evaluation of mental health conditions such as depression are still performed by professional psychologists. The data collected and integrated by the emotional state assessment system and the preliminary judgment of the emotional state will be used as auxiliary tools to assist professional physicians in making more accurate decisions during the diagnosis process.

[0090] In actual application, the emotional state assessment system can also provide more detailed emotional fluctuation information through continuous emotional state monitoring, so as to more accurately identify the chronic development process of mental illnesses such as depression. The specific steps are as follows:

[0091] Within a preset time period, multiple emotional states of the user to be tested within the preset time period can be continuously obtained; and based on the multiple emotional states within the preset time period, a long short-term memory network is used to construct an emotional state time series model of the user to be tested, and the changing trend of the emotional state of the user to be tested is judged based on the emotional state time series model.

[0092] That is to say, in actual application, if the emotional state assessment system designed by the present invention continues to detect that the user's emotional state is gradually deteriorating within a preset time period, for example: in continuous weeks of monitoring, it is found that the user's emotional state is gradually becoming depressed, the speech speed is slowed down, the facial expressions are reduced, the facial heart rate is reduced, etc., then the emotional state assessment system can give a preliminary depression risk assessment.

[0093] In summary, the emotional state assessment system architecture designed by the present invention is highly open, allowing new modal data or analysis methods to be integrated. With the development of science and technology, new physiological or behavioral data sources (such as skin galvanic response, brain waves) can be seamlessly integrated into the emotional state assessment system, which makes the system highly extensible and adaptable to the future, and maintains its position at the forefront of technology.

[0094] At the same time, in clinical applications, the multimodal emotional state assessment method and system designed by the present invention can not only assist psychologists in making accurate diagnoses, but also provide real-time emotional state assessment in telemedicine, which is especially important for areas lacking medical resources.

[0095] In addition, in commercial applications, the multimodal emotional state assessment method and system designed by the present invention can be used to develop a personalized mental health service platform to provide users with continuous mental state monitoring, and provide corresponding health advice and intervention measures, thereby opening up new market opportunities. It has good promotion prospects and application value.

[0096] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal emotional state assessment method, characterized in that: Includes steps: S1: Collecting multimodal data of the user to be tested, wherein the multimodal data includes voice data, facial micro-expression data, and facial heart rate data; S2: construct a multimodal deep neural network, input the multimodal data into the multimodal deep neural network, and the multimodal deep neural network dynamically adjusts the weight of the modal data using an attention mechanism to output a fusion feature; S3: Determine the emotional state of the user to be tested based on the fusion features.

2. The multimodal emotional state assessment method according to claim 1, characterized in that: In step S2, specifically: S21: construct a multimodal deep neural network, and pre-set initial weights for each modal data based on the statistical analysis results of historical sample data; S22: inputting the multimodal data into the multimodal deep neural network, and when detecting that a feature change of a certain modal data reflects the emotional state more significantly, the multimodal deep neural network automatically increases the weight of the corresponding modal data using a trained weight adjustment model; S23: The multimodal deep neural network outputs fusion features.

3. The multimodal emotional state assessment method according to claim 2, characterized in that: In step S23, the weight adjustment model automatically increases the weight of the corresponding modal data based on the context factor, the historical data factor and the immediate feedback factor.

4. The multimodal emotional state assessment method according to claim 1, characterized in that: In step S2, the fusion features include: sound audio features, voice and text features, facial image features and facial physiological features.

5. The multimodal emotional state assessment method according to claim 4, characterized in that: In step S3, a comprehensive judgment of the emotional state is performed based on the fusion features, specifically: Based on the sound audio features and speech and text features, the speech speed, sound intonation and content are analyzed to determine the emotional state of the user to be tested; detailed analysis is performed based on the facial image features to determine the instantaneous emotional state of the user to be tested; and the facial heart rate changes of the user to be tested are analyzed through the facial physiological features to determine the emotional state of the user to be tested.

6. The multimodal emotional state assessment method according to claim 1, characterized in that: The step S4 is also included: Within a preset time period, S1-S3 are repeated to obtain a plurality of emotional states of the user to be tested within the preset time period, and to determine a change trend of the emotional state of the user to be tested.

7. The multimodal emotional state assessment method according to claim 1, characterized in that: In step S4, based on the multiple emotional states within the preset time period, a long short-term memory network is used to construct an emotional state time series model of the user to be tested, and the change trend of the emotional state of the user to be tested is determined based on the emotional state time series model.

8. A multimodal emotional state assessment system, applied to the multimodal emotional state assessment method according to any one of claims 1 to 7, characterized in that: include: A multimodal interaction module, a data receiving module and a multi-level analysis module, wherein the multimodal interaction module is communicatively connected to the data receiving module, and the data receiving module is communicatively connected to the multi-level analysis module; The multimodal interaction module is used to collect multimodal data, including voice data, facial micro-expression data and facial heart rate data, and send the data to the data receiving module; the data receiving module integrates the multimodal data and generates corresponding fusion features; The multi-level analysis module evaluates the emotional state based on the fused features.

9. The multimodal emotional state assessment system according to claim 8, characterized in that: The multimodal interaction module includes: an audio capture unit, an image capture unit and a heart rate monitoring unit, and the audio capture unit, the image capture unit and the heart rate monitoring unit collect voice data, facial micro-expression data and facial heart rate data respectively.

10. The multimodal emotional state assessment system according to claim 8, characterized in that: The data receiving module uses a multimodal deep neural network with an attention mechanism to integrate the multimodal data and generate corresponding fusion features.

Images