Multi-mode emotion micro-fluctuation tracking method and device for anti-depression robot
By using multimodal data fusion and personalized emotion models, the shortcomings of existing technologies in monitoring the emotions of patients with depression have been addressed. This enables timely identification and personalized intervention of subtle emotional fluctuations, thereby improving the effectiveness of emotion monitoring and intervention for patients with depression.
Patent Information
- Application Number
- CN202511396334.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-28
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies lack multimodal data fusion and personalized adaptation in monitoring the emotions of patients with depression, resulting in poor intervention effects and difficulty in capturing subtle emotional fluctuations and intervening in a timely manner.
By collecting facial expression images, voice tone fragments, and physiological indicator data, a multi-source dataset is constructed. Feature extraction and fusion are performed to generate emotional fluctuation index values. Combined with a personalized emotion model, emotional state assessment is conducted, and emotion regulation suggestion signals are generated and customized intervention content is pushed out.
It enables timely identification and personalized intervention of subtle mood fluctuations in patients with depression, improving the accuracy of mood monitoring and the effectiveness of intervention.
Smart Images
Figure CN121489477A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of emotion recognition, and in particular to a multi-modal emotion micro-fluctuation tracking method and device for an anti-depression robot. BACKGROUND
[0002] In the field of mental health, early intervention and emotion management for depression are considered as key directions to improve the quality of life of patients and prevent the disease from worsening. With the advancement of technology, intelligent devices have shown important potential in assisting mental health monitoring and intervention, especially in emotion recognition and real-time tracking.
[0003] Currently, methods for monitoring the emotions of depression patients rely on a single data source, such as questionnaires or simple behavior observation, which often fail to capture subtle signals of emotional changes and miss the best opportunity for intervention. At the same time, existing technologies lack personalized adaptation capabilities in the face of individual differences, resulting in a significant reduction in the effectiveness of intervention measures.
[0004] How to capture the small fluctuations in the emotions of depression patients in real time and intervene is a technical problem that needs to be overcome by those skilled in the art. SUMMARY
[0005] The present application provides a multi-modal emotion micro-fluctuation tracking method and device for an anti-depression robot to at least partially solve the above technical problems.
[0006] In a first aspect, to solve the above technical problems, the present application provides a multi-modal emotion micro-fluctuation tracking method for an anti-depression robot, comprising: Collecting facial expression images, sound tone segments, and physiological indicator monitoring data of depression patients to construct an initial multi-source data set; Performing feature extraction analysis on the facial expression images to obtain preliminary emotion fluctuation characteristic values of micro-expression changes in frequency and amplitude; Combining the preliminary emotion fluctuation characteristic values and sound tone segments to generate emotion fluctuation indicator values; Identifying a dynamic pattern of emotional change trends based on the emotion fluctuation indicator values and physiological indicator monitoring data; Adjusting the basic emotion model based on the dynamic pattern of emotional change trends to obtain a personalized emotion model; obtaining an emotional state assessment result based on the personalized emotion model and the multi-source data in the multi-source data set; If the emotion fluctuation indicator value in the emotional state assessment result exceeds the preset indicator value, an emotion regulation suggestion signal is generated; Controlling the intelligent device to push customized intervention content according to the emotion regulation suggestion signal.
[0007] In an alternative embodiment, feature extraction analysis of the facial expression image is performed to obtain preliminary emotional fluctuation characteristic values, including: The facial expression image is input into a pre-trained expression recognition model; the pre-trained expression recognition model extracts expression-related feature data in each frame of the facial expression image to generate an initial expression feature set; Time series analysis is performed on the feature vectors in the initial expression feature set to calculate the Euclidean distance between adjacent frames; If the Euclidean distance exceeds a set distance threshold, it is determined to be a micro-expression change, and the number of micro-expression changes per unit time is counted as the frequency of micro-expression changes; The maximum value of the Euclidean distance during the micro-expression change is calculated as the amplitude of the micro-expression change; The preliminary emotional fluctuation characteristic values are calculated in combination with the frequency and amplitude.
[0008] In an alternative embodiment, the preliminary emotional fluctuation characteristic values and the voice tone segment are combined to generate emotional fluctuation index values, including: Based on the voice tone segment, the speech rate, tone, and volume are extracted; the speech rate, tone, and volume are speech feature parameters; If the change amplitude of any speech feature parameter exceeds the corresponding change threshold, the parameter is weighted to obtain an emotional feature value group; the emotional feature value group includes a speech rate weighted feature value, a tone weighted feature value, and a volume weighted feature value; The emotional feature value group is fused to generate an emotional fluctuation vector; The generated emotional fluctuation vector is input into a support vector machine model and matched with a pre-defined emotional fluctuation pattern; the support vector machine model outputs a corresponding emotional fluctuation category; According to a pre-established emotional index mapping table, the emotional fluctuation category is converted into an emotional fluctuation index value.
[0009] In an alternative embodiment, based on the emotional fluctuation index value and physiological indicator monitoring data, the dynamic mode of emotional change trend is identified, including: The emotional fluctuation index value and physiological indicator data are time-synchronized to form a unified time series data set; the physiological indicator data includes heart rate signals and skin galvanic response signals; The unified time series data set is modeled using a time series analysis method to extract an emotional change trend curve; if the fluctuation amplitude of the trend curve exceeds a pre-set fluctuation threshold, the curve is smoothed; Based on the sentiment change trend curve and the weight allocation of comprehensive indicators, the characteristic parameters of the dynamic trend are calculated; the characteristic parameters of the dynamic trend include: trend slope, fluctuation amplitude, periodic component and peak interval. Based on the feature parameters of dynamic trends, the support vector machine algorithm is used to classify the trend of emotion change and determine the category of the current emotion change. Based on the classification results and the periodic characteristics of the time series, a dynamic pattern of emotion change trends is obtained.
[0010] In one optional implementation, a personalized emotion model is obtained by adjusting the parameters of the basic emotion model based on the dynamic pattern of the emotion change trend, including: S501. Extract feature parameters related to the dynamic pattern of emotion change trends. The related feature parameters include the frequency of micro-changes in facial expressions, voice feature parameters, and feature parameters of dynamic trends to form an initial feature matrix. S502. Input the initial feature matrix into the constructed basic sentiment model, and calculate the correlation between the feature parameters and historical sentiment labels through a sliding time window to obtain the weight coefficient of each feature parameter. S503. Based on the dynamic pattern of emotion change trends, the gradient descent algorithm is used to iteratively optimize the weight coefficients; if the correlation coefficient between a certain feature and emotion change is higher than the preset value in multiple consecutive windows, the weight of that feature is increased. S504. Introduce an individual difference calibration factor; the calibration factor is calculated based on the patient's emotional feedback data over a period of time. S505. The deviation between the model output and the actual emotional state is evaluated by using a confusion matrix. When the deviation rate exceeds a preset deviation value, secondary feature extraction is triggered to supplement auxiliary features in the relevant feature parameters. The auxiliary features include the sentiment tendency of social media texts or sleep duration. Repeat steps S502-S505 until the model's emotion prediction accuracy on the validation set reaches the preset standard, thus obtaining a personalized emotion model.
[0011] In one optional implementation, if the emotional fluctuation index value in the emotional state assessment result exceeds a preset index value, an emotional regulation suggestion signal is generated, including: The emotion classification results are constructed by combining the emotion state assessment results output by the personalized emotion model with the user's historical behavior data and real-time collected physiological indicator monitoring signals. The fluctuation index of the emotion classification results is calculated in real time. If the calculated fluctuation index exceeds the preset threshold, the state detection module is triggered to identify whether there is abnormal emotion fluctuation. The status detection module acquires the changing trend of fluctuation indicators and analyzes the changing trend to determine whether there is a persistent emotional abnormality. If persistent emotional abnormalities are determined, the corresponding intervention mechanism configuration is retrieved from the emotion regulation strategy database to determine the appropriate real-time intervention strategy. Based on the real-time intervention strategy configuration information, targeted emotion regulation suggestions are generated, and the emotion regulation suggestions are formatted using a preset suggestion template to generate regulation suggestion signals. The regulation suggestion signals include recommending the playback of soothing music at a specific frequency, pushing text prompts, and guiding users to perform deep breathing exercises.
[0012] In an optional implementation, the method further includes: The adjustment suggestion signal is converted into an executable instruction set based on a pre-established signal parsing module; The instruction set is output to the smart device interface and transmitted to the target user terminal in the form of intervention instructions to be pushed, thereby transmitting personalized intervention content to the target user terminal; After receiving the intervention content, the system collects the user's emotional fluctuation feedback data in real time; the emotional fluctuation feedback data is cleaned to form structured emotional fluctuation data, and the system determines whether the integrity of the emotional fluctuation data meets the preset threshold. If the completeness of the structured emotion fluctuation data reaches a preset threshold, then feature extraction is performed on the emotion fluctuation data to output key indicators of emotion changes. By combining a pre-established model iteration framework, the parameters of the personalized emotion model are adjusted according to key indicators of emotion changes to generate an updated model version, and the performance of the model is evaluated to see if it has improved. If the model performance improves, then update the personalized emotion model.
[0013] In an optional implementation, the method further includes: Construct a multi-scenario intervention strategy mapping table; the mapping table contains the correspondence between different environment categories and appropriate intervention methods; Identify the current environment type described by the user; Prioritize the intervention method based on the environment type.
[0014] In a second aspect, the present invention provides a multimodal emotion micro-fluctuation tracking device for antidepressant robots, comprising: The first processing module is used to: collect facial expression images, voice tone fragments, and physiological indicator monitoring data of patients with depression, and construct an initial multi-source dataset; The second processing module is used to: extract and analyze the facial expression image to obtain preliminary emotional fluctuation feature values by analyzing the frequency and amplitude of micro-expression changes. The third processing module is used to: combine the preliminary emotional fluctuation feature values and voice tone segments to generate an emotional fluctuation index value; The fourth processing module is used to: identify dynamic patterns of emotional change trends based on the emotional fluctuation index values and physiological index monitoring data; The fifth processing module is used to: adjust the parameters of the basic emotion model according to the dynamic pattern of the emotion change trend to obtain a personalized emotion model; and obtain the emotion state assessment result based on the personalized emotion model and the multi-source data in the multi-source dataset. The sixth processing module is used to generate an emotion regulation suggestion signal if the emotion fluctuation index value in the emotion state assessment result exceeds the preset index value. The seventh processing module is used to control the smart device to push customized intervention content based on the emotion regulation suggestion signal.
[0015] Compared with existing technologies, the present invention has at least the following beneficial effects: the ability to identify subtle mood fluctuations in patients with depression facilitates timely intervention. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating the multimodal emotion micro-fluctuation tracking method for an antidepressant robot provided in the first embodiment of the present invention; Figure 2 This is a block diagram of a multimodal emotion micro-fluctuation tracking device for an antidepressant robot provided in the second embodiment of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] Reference Figure 1 The first embodiment of the present invention provides a multimodal emotion micro-fluctuation tracking method for antidepressant robots, comprising the following steps: S101: Collect facial expression images, voice and tone fragments, and physiological indicator monitoring data of patients with depression to construct an initial multi-source dataset; S102, Perform feature extraction and analysis on the facial expression image to obtain preliminary emotional fluctuation feature values by analyzing the frequency and amplitude of micro-expression changes; S103, Combine the preliminary emotional fluctuation feature values and voice tone segments to generate an emotional fluctuation index value; S104, Identify dynamic patterns of emotional change trends based on the emotional fluctuation index values and physiological index monitoring data; S105, Based on the dynamic pattern of the emotion change trend, the basic emotion model is adjusted with personalized parameters to obtain a personalized emotion model; based on the personalized emotion model and the multi-source data in the multi-source dataset, the emotion state assessment result is obtained. S106, if the emotional fluctuation index value in the emotional state assessment result exceeds the preset index value, an emotional regulation suggestion signal is generated.
[0019] S107, Based on the emotion regulation suggestion signal, control the smart device to push customized intervention content.
[0020] In one embodiment, feature extraction and analysis of the facial expression image is performed to obtain preliminary emotional fluctuation feature values by analyzing the frequency and amplitude of micro-expression changes. Facial expression images are input into a pre-trained facial expression recognition model; the pre-trained facial expression recognition model extracts expression-related feature data from each frame of facial expression images to generate an initial set of facial expression features; Time series analysis is performed on the feature vectors in the initial set of facial expression features to calculate the Euclidean distance between adjacent frames; If the Euclidean distance exceeds the set distance threshold, it is determined as a micro-expression change, and the number of micro-expression changes per unit time is counted as the frequency of micro-expression changes. The maximum value of the Euclidean distance during the micro-expression change is calculated as the amplitude of the micro-expression change; Preliminary emotional fluctuation characteristic values are calculated by combining the frequency and amplitude.
[0021] Specifically, facial expression images captured by a high-definition camera (30 frames per second) can be input into a pre-trained deep learning-based facial expression recognition model, such as the ResNet-50 model. This model has been trained on the FER2013 dataset and can recognize seven basic emotions (anger, disgust, fear, happiness, sadness, surprise, and neutrality). The facial expression recognition model outputs feature vectors for each frame of the image through its feature extraction layer, forming an initial set of facial expression features.
[0022] For the feature vectors in the initial set of facial expression features, time series analysis is used to evaluate the similarity between adjacent frames. Specifically, the Euclidean distance between the feature vectors of adjacent frames is calculated to quantify the degree of change in facial expressions between consecutive frames. A threshold (e.g., 0.5) can be set; if the calculated distance exceeds this threshold, a micro-expression change is considered to have occurred.
[0023] Within a unit of time (e.g., per minute), the number of times a change is identified as a micro-expression is recorded as the frequency of micro-expression changes. The frequency of micro-expression changes reflects the activity level of an individual's facial expression changes within a specific time period. The maximum Euclidean distance of all identified micro-expression changes is recorded as the amplitude of the micro-expression change.
[0024] Based on the frequency and amplitude information obtained above, a preliminary characteristic value of emotional fluctuation is calculated using a weighted average formula. For example, the final characteristic value can be obtained using the formula: Characteristic Value = 0.6 × Frequency + 0.4 × Amplitude, thus reflecting the individual's emotional fluctuation.
[0025] In one implementation, the emotional fluctuation index value is generated by fusing the preliminary emotional fluctuation feature value and the vocal tone segment, including: Speech rate, pitch, and volume are extracted from the aforementioned speech segments; speech rate, pitch, and volume are recorded as speech feature parameters. If the change in any speech feature parameter exceeds the corresponding change threshold, the parameter is weighted to obtain an emotion feature value group; the emotion feature value group includes speech rate weighted feature value, pitch weighted feature value, and volume weighted feature value; The emotion feature value groups are fused to generate an emotion fluctuation vector; The generated emotion fluctuation vector is input into the support vector machine model and matched with a predefined emotion fluctuation pattern; the support vector machine model outputs the corresponding emotion fluctuation category. Based on a pre-established emotion index mapping table, the emotion fluctuation categories are converted into emotion fluctuation index values.
[0026] Specifically, vocal segments of the patient were recorded using a high-sensitivity microphone at a sampling rate of 44.1 kHz, and speech rate, pitch, and volume were extracted using acoustic analysis tools. If the variation of any speech feature parameter exceeded a preset threshold, the parameter was weighted to obtain a set of emotional feature values. For example, if the pitch variation exceeded the normal range (e.g., 200-300 Hz), it was given a higher weight based on its deviation, thus forming a feature set including speech rate-weighted feature values, pitch-weighted feature values, and volume-weighted feature values. A weighted averaging algorithm was used to combine these weighted emotional feature values into an emotional fluctuation vector. The generated emotional fluctuation vector was input into a pre-trained support vector machine (SVM) model and matched against predefined emotional fluctuation patterns. Based on a pre-established emotional index mapping table, the emotional fluctuation categories output by the SVM model were converted into specific emotional fluctuation index values. This mapping table transforms abstract emotional fluctuation categories into quantifiable numerical values.
[0027] The above approach extracts preliminary emotional fluctuation features from facial expression images of patients with depression. These features are then combined with key speech feature parameters such as speech rate, pitch, and volume from speech segments. After weighted processing, a more comprehensive emotional fluctuation vector is generated. Because single-modal data often fails to fully reflect an individual's emotional changes, this application significantly improves the accuracy of identifying the patient's emotional state.
[0028] In one implementation, identifying dynamic patterns of emotional change trends based on the emotional fluctuation index values and physiological indicator monitoring data includes: The emotional fluctuation index values and physiological index data are time-synchronized to form a unified time-series data set; the physiological index data includes heart rate signals and skin conductance response signals. The unified time series data set is modeled using time series analysis methods to extract the sentiment change trend curve; if the fluctuation amplitude of the trend curve exceeds a preset fluctuation threshold, the curve is smoothed. Based on the sentiment change trend curve and the weight allocation of comprehensive indicators, the characteristic parameters of the dynamic trend are calculated; the characteristic parameters of the dynamic trend include: trend slope, fluctuation amplitude, periodic component and peak interval. Based on the feature parameters of dynamic trends, the support vector machine algorithm is used to classify the trend of emotion change and determine the category of the current emotion change. Based on the classification results and the periodic characteristics of the time series, a dynamic pattern of emotion change trends is obtained.
[0029] Specifically, emotional fluctuation index values and physiological index data are synchronized over time to ensure that data from different sources can be compared and analyzed within the same time frame, thus forming a unified time series dataset. Time series analysis methods are used to model this unified time series dataset to extract the trend curve of emotional changes. If the fluctuation amplitude of this trend curve exceeds a preset fluctuation threshold, the curve needs to be smoothed to reduce noise interference. Based on the emotional change trend curve and combined with the weight allocation of comprehensive indicators, key characteristic parameters of the dynamic trend are calculated. These characteristic parameters may include trend slope, fluctuation amplitude, periodic component, and peak interval. The trend slope represents the speed or direction of emotional change, the fluctuation amplitude describes the degree of emotional change, the periodic component refers to the recurring part of emotional change, and the peak interval refers to the time distance between two adjacent emotional peaks. The support vector machine algorithm is used to classify the emotional change trend according to the characteristic parameters of the dynamic trend to determine which category the current emotional change belongs to. Based on the classification results and combined with the periodic characteristics of the time series, the dynamic pattern of the emotional change trend is determined.
[0030] In one implementation, a personalized emotion model is obtained by adjusting the parameters of a basic emotion model based on the dynamic pattern of the emotion change trend, including: S501. Extract feature parameters related to the dynamic pattern of emotion change trends. The related feature parameters include the frequency of micro-changes in facial expressions, voice feature parameters, and feature parameters of dynamic trends to form an initial feature matrix. S502. Input the initial feature matrix into the constructed basic sentiment model, and calculate the correlation between the feature parameters and historical sentiment labels through a sliding time window to obtain the weight coefficient of each feature parameter. S503. Based on the dynamic pattern of emotion change trends, the gradient descent algorithm is used to iteratively optimize the weight coefficients; if the correlation coefficient between a certain feature and emotion change is higher than the preset value in multiple consecutive windows, the weight of that feature is increased. S504. Introduce an individual difference calibration factor; the calibration factor is calculated based on the patient's emotional feedback data over a period of time. S505. The deviation between the model output and the actual emotional state is evaluated by using a confusion matrix. When the deviation rate exceeds a preset deviation value, secondary feature extraction is triggered to supplement auxiliary features in the relevant feature parameters. The auxiliary features include the sentiment tendency of social media texts or sleep duration. Repeat steps S502-S505 until the model's emotion prediction accuracy on the validation set reaches the preset standard, thus obtaining a personalized emotion model.
[0031] Specifically, feature parameters related to the dynamic pattern of emotion change trends are extracted, including the frequency of subtle changes in facial expressions, voice feature parameters, and dynamic trend feature parameters. The initial feature matrix is a data structure containing multiple emotion-related features. This initial feature matrix is input into the constructed basic emotion model, and the correlation between feature parameters and historical emotion labels is calculated using a sliding time window technique to determine the weight coefficients of each feature parameter. Based on the dynamic pattern of emotion change trends, a gradient descent algorithm is used to iteratively optimize the weight coefficients. If the correlation coefficient between a certain feature and emotion change is higher than a preset value within multiple consecutive windows, the weight of that feature is increased, ensuring that the model can more accurately reflect the individual's emotional change characteristics. An individual difference calibration factor is calculated based on the patient's past emotional feedback data. This factor aims to compensate for differences in emotional expression between different individuals, making the model more universal. A confusion matrix is used to assess the deviation between the model output and the actual emotional state. When the deviation rate exceeds a preset deviation value, secondary feature extraction is triggered, supplementing relevant feature parameters with auxiliary features, such as the sentiment tendency of social media texts or sleep duration. Repeat steps S502 to S505 until the model's emotion prediction accuracy on the validation set reaches the preset standard, ultimately obtaining a personalized emotion model suitable for a specific individual.
[0032] In one implementation, if the emotional fluctuation index value in the emotional state assessment result exceeds a preset index value, an emotional regulation suggestion signal is generated, including: The emotion classification results are constructed by combining the emotion state assessment results output by the personalized emotion model with the user's historical behavior data and real-time collected physiological indicator monitoring signals. The fluctuation index of the emotion classification results is calculated in real time. If the calculated fluctuation index exceeds the preset threshold, the state detection module is triggered to identify whether there is abnormal emotion fluctuation. The status detection module acquires the changing trend of fluctuation indicators and analyzes the changing trend to determine whether there is a persistent emotional abnormality. If persistent emotional abnormalities are determined, the corresponding intervention mechanism configuration is retrieved from the emotion regulation strategy database to determine the appropriate real-time intervention strategy. Based on the real-time intervention strategy configuration information, targeted emotion regulation suggestions are generated, and the emotion regulation suggestions are formatted using a preset suggestion template to generate regulation suggestion signals. The regulation suggestion signals include recommending the playback of soothing music at a specific frequency, pushing text prompts, and guiding users to perform deep breathing exercises.
[0033] Specifically, the system uses a personalized emotion model to obtain emotional state assessment results from users' historical behavioral data (such as social media interactions and daily activity records) and real-time collected physiological indicator monitoring signals, and combines this information to construct emotion classification results. The emotion classification results refer to the categorization of emotions based on the user's current emotional state. A fluctuation index is calculated in real time based on the emotion classification results. If the calculated fluctuation index exceeds a preset threshold, a state detection module is triggered to identify whether abnormal emotional fluctuations exist. The state detection module acquires and analyzes the changing trends of the fluctuation index: once the state detection module is triggered, the system acquires the changing trends of the fluctuation index and further analyzes them to determine whether persistent emotional abnormalities have occurred. If persistent emotional abnormalities are determined, the system automatically retrieves the corresponding intervention mechanism configuration from the emotion regulation strategy database to determine the appropriate real-time intervention strategy; the emotion regulation strategy database contains various effective intervention measures for different emotional states.
[0034] Based on real-time intervention strategy configuration information, targeted emotion regulation suggestions are generated. These suggestions are then formatted using a preset suggestion template to generate regulation suggestion signals. For example, regulation suggestion signals might include recommending the playback of soothing music at a specific frequency (e.g., 432Hz), sending text prompts (e.g., instructions to guide users in deep breathing exercises), and providing specific exercise guidance (e.g., maintaining a breathing rate of 6 breaths per minute). These measures aim to help users quickly alleviate negative emotions and restore mental health.
[0035] In one embodiment, the method further includes: The adjustment suggestion signal is converted into an executable instruction set based on a pre-established signal parsing module; The instruction set is output to the smart device interface and transmitted to the target user terminal in the form of intervention instructions to be pushed, thereby transmitting personalized intervention content to the target user terminal; After receiving the intervention content, the system collects the user's emotional fluctuation feedback data in real time; the emotional fluctuation feedback data is cleaned to form structured emotional fluctuation data, and the system determines whether the integrity of the emotional fluctuation data meets the preset threshold. If the completeness of the structured emotion fluctuation data reaches a preset threshold, then feature extraction is performed on the emotion fluctuation data to output key indicators of emotion changes. By combining a pre-established model iteration framework, the parameters of the personalized emotion model are adjusted according to key indicators of emotion changes to generate an updated model version, and the performance of the model is evaluated to see if it has improved. If the model performance improves, then update the personalized emotion model.
[0036] Specifically, to further optimize the personalized emotion model, a pre-established signal parsing module converts adjustment suggestion signals into an executable instruction set, which is then output to a smart device interface (a smart device refers to a device connecting the system and the user terminal, such as a smartphone or smartwatch). This instruction set is transmitted to the target user terminal as an intervention instruction to be pushed, thus transmitting the personalized intervention content to the target user terminal. After receiving the intervention content, the user's emotional fluctuation feedback data is collected in real time through the sensor interface of the user terminal. This emotional fluctuation feedback data is cleaned to form structured emotional fluctuation data, and the completeness of the emotional fluctuation data is assessed to determine if it meets a preset threshold. If the completeness of the structured emotional fluctuation data meets the preset threshold, feature extraction is performed on the emotional fluctuation data, and key indicators of emotional change are output. Combined with a pre-established model iteration framework, the parameters of the personalized emotion model are adjusted according to the key indicators of emotional change to generate an updated model version, and the model performance is evaluated to see if there is any improvement. If the model performance is improved, the personalized emotion model is updated.
[0037] In one embodiment, the method further includes: Construct a multi-scenario intervention strategy mapping table; the mapping table contains the correspondence between different environment categories and appropriate intervention methods; Identify the current environment type described by the user; Prioritize the intervention method based on the environment type.
[0038] Specifically, a multi-scenario intervention strategy mapping table is constructed. The intervention strategy mapping table contains the correspondence between different environmental categories (environmental categories can be the user's geographical location, activity status, or time, etc.) and appropriate intervention methods (appropriate intervention methods include playing specific types of music and pushing positive psychological cues).
[0039] Prioritizing intervention methods based on environment type means searching for corresponding intervention methods in a mapping table based on the identified environment type and setting priorities for these methods. Once the user's environment type is determined, the system will configure the priority of intervention methods accordingly. In this way, the system can dynamically adjust intervention measures to better suit the user's actual needs and environmental characteristics.
[0040] Reference Figure 2 The second embodiment of the present invention provides a multimodal emotion micro-fluctuation tracking device for an antidepressant robot, comprising: The first processing module is used to: collect facial expression images, voice tone fragments, and physiological indicator monitoring data of patients with depression, and construct an initial multi-source dataset; The second processing module is used to: extract and analyze the facial expression image to obtain preliminary emotional fluctuation feature values by analyzing the frequency and amplitude of micro-expression changes. The third processing module is used to: combine the preliminary emotional fluctuation feature values and voice tone segments to generate an emotional fluctuation index value; The fourth processing module is used to: identify dynamic patterns of emotional change trends based on the emotional fluctuation index values and physiological index monitoring data; The fifth processing module is used to: adjust the parameters of the basic emotion model according to the dynamic pattern of the emotion change trend to obtain a personalized emotion model; and obtain the emotion state assessment result based on the personalized emotion model and the multi-source data in the multi-source dataset. The sixth processing module is used to generate an emotion regulation suggestion signal if the emotion fluctuation index value in the emotion state assessment result exceeds the preset index value. The seventh processing module is used to control the smart device to push customized intervention content based on the emotion regulation suggestion signal.
[0041] It should be noted that the multimodal emotion micro-fluctuation tracking device for antidepressant robots provided in this embodiment of the invention is used to execute all the process steps of the multimodal emotion micro-fluctuation tracking method for antidepressant robots in the above embodiments. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.
[0042] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.
Claims
1. A multimodal emotion micro-fluctuation tracking method for antidepressant robots, characterized in that, include: We collected facial expression images, vocal tone fragments, and physiological indicator monitoring data from patients with depression to construct an initial multi-source dataset. Feature extraction and analysis of the facial expression images were performed to obtain preliminary emotional fluctuation feature values by analyzing the frequency and amplitude of micro-expression changes. The emotional fluctuation index value is generated by combining the preliminary emotional fluctuation feature values and voice tone segments. Based on the aforementioned emotional fluctuation index values and physiological index monitoring data, dynamic patterns of emotional change trends are identified. A personalized emotion model is obtained by adjusting the parameters of the basic emotion model based on the dynamic pattern of the emotion change trend. Emotional state assessment results are obtained based on personalized emotion models and multi-source data in multi-source datasets. If the emotional fluctuation index value in the emotional state assessment result exceeds the preset index value, an emotional regulation suggestion signal is generated. Based on the aforementioned emotion regulation suggestion signals, the smart device is controlled to push customized intervention content.
2. The multimodal emotion micro-fluctuation tracking method for antidepressant robots according to claim 1, characterized in that, Feature extraction and analysis of the facial expression images reveals the frequency and amplitude of micro-expression changes, yielding preliminary emotional fluctuation feature values, including: Facial expression images are input into a pre-trained facial expression recognition model; the pre-trained facial expression recognition model extracts expression-related feature data from each frame of facial expression images to generate an initial set of facial expression features; Time series analysis is performed on the feature vectors in the initial set of facial expression features to calculate the Euclidean distance between adjacent frames; If the Euclidean distance exceeds the set distance threshold, it is determined as a micro-expression change, and the number of micro-expression changes per unit time is counted as the frequency of micro-expression changes. The maximum value of the Euclidean distance during the micro-expression change is calculated as the amplitude of the micro-expression change; Preliminary emotional fluctuation characteristic values are calculated by combining the frequency and amplitude.
3. The multimodal emotion micro-fluctuation tracking method for antidepressant robots according to claim 2, characterized in that, The emotional fluctuation index value is generated by fusing the preliminary emotional fluctuation feature values and vocal tone segments, including: Speech rate, pitch, and volume are extracted from the aforementioned speech segments; speech rate, pitch, and volume are recorded as speech feature parameters. If the change in any speech feature parameter exceeds the corresponding change threshold, the parameter is weighted to obtain an emotion feature value group; the emotion feature value group includes speech rate weighted feature value, pitch weighted feature value, and volume weighted feature value; The emotion feature value groups are fused to generate an emotion fluctuation vector; The generated emotion fluctuation vector is input into the support vector machine model and matched with a predefined emotion fluctuation pattern; the support vector machine model outputs the corresponding emotion fluctuation category. Based on a pre-established emotion index mapping table, the emotion fluctuation categories are converted into emotion fluctuation index values.
4. The multimodal emotion micro-fluctuation tracking method for antidepressant robots according to claim 3, characterized in that, Dynamic patterns of emotion change trends are identified based on the aforementioned emotion fluctuation index values and physiological indicator monitoring data, including: The emotional fluctuation index values and physiological index data are time-synchronized to form a unified time-series data set; the physiological index data includes heart rate signals and skin conductance response signals. The unified time series data set is modeled using time series analysis methods to extract the sentiment change trend curve; if the fluctuation amplitude of the trend curve exceeds a preset fluctuation threshold, the curve is smoothed. Based on the sentiment change trend curve and the weight allocation of comprehensive indicators, the characteristic parameters of the dynamic trend are calculated; the characteristic parameters of the dynamic trend include: trend slope, fluctuation amplitude, periodic component and peak interval. Based on the feature parameters of dynamic trends, the support vector machine algorithm is used to classify the trend of emotion change and determine the category of the current emotion change. Based on the classification results and the periodic characteristics of the time series, a dynamic pattern of emotion change trends is obtained.
5. The multimodal emotion micro-fluctuation tracking method for antidepressant robots according to claim 4, characterized in that, A personalized emotion model is obtained by adjusting the parameters of the basic emotion model based on the dynamic pattern of the emotion change trend, including: S501. Extract feature parameters related to the dynamic pattern of emotion change trends. The related feature parameters include the frequency of micro-changes in facial expressions, voice feature parameters, and feature parameters of dynamic trends to form an initial feature matrix. S502. Input the initial feature matrix into the constructed basic sentiment model, and calculate the correlation between the feature parameters and historical sentiment labels through a sliding time window to obtain the weight coefficient of each feature parameter. S503. Based on the dynamic pattern of emotion change trends, the gradient descent algorithm is used to iteratively optimize the weight coefficients; if the correlation coefficient between a certain feature and emotion change is higher than the preset value in multiple consecutive windows, the weight of that feature is increased. S504. Introduce an individual difference calibration factor; the calibration factor is calculated based on the patient's emotional feedback data over a period of time. S505. The deviation between the model output and the actual emotional state is evaluated by using a confusion matrix. When the deviation rate exceeds a preset deviation value, secondary feature extraction is triggered to supplement auxiliary features in the relevant feature parameters. The auxiliary features include the sentiment tendency of social media texts or sleep duration. Repeat steps S502-S505 until the model's emotion prediction accuracy on the validation set reaches the preset standard, thus obtaining a personalized emotion model.
6. The multimodal emotion micro-fluctuation tracking method for antidepressant robots according to claim 5, characterized in that, If the emotional fluctuation index value in the emotional state assessment result exceeds the preset index value, an emotional regulation suggestion signal is generated, including: The emotion classification results are constructed by combining the emotion state assessment results output by the personalized emotion model with the user's historical behavior data and real-time collected physiological indicator monitoring signals. The fluctuation index of the emotion classification results is calculated in real time. If the calculated fluctuation index exceeds the preset threshold, the state detection module is triggered to identify whether there is abnormal emotion fluctuation. The status detection module acquires the changing trend of fluctuation indicators and analyzes the changing trend to determine whether there is a persistent emotional abnormality. If persistent emotional abnormalities are determined, the corresponding intervention mechanism configuration is retrieved from the emotion regulation strategy database to determine the appropriate real-time intervention strategy. Based on the real-time intervention strategy configuration information, targeted emotion regulation suggestions are generated, and the emotion regulation suggestions are formatted using a preset suggestion template to generate regulation suggestion signals. The regulation suggestion signals include recommending the playback of soothing music at a specific frequency, pushing text prompts, and guiding users to perform deep breathing exercises.
7. The multimodal emotion micro-fluctuation tracking method for antidepressant robots according to claim 6, characterized in that, The method further includes: The adjustment suggestion signal is converted into an executable instruction set based on a pre-established signal parsing module; The instruction set is output to the smart device interface and transmitted to the target user terminal in the form of intervention instructions to be pushed, thereby transmitting personalized intervention content to the target user terminal; After receiving the intervention content, the system collects the user's emotional fluctuation feedback data in real time; the emotional fluctuation feedback data is cleaned to form structured emotional fluctuation data, and the system determines whether the integrity of the emotional fluctuation data meets the preset threshold. If the completeness of the structured emotion fluctuation data reaches a preset threshold, then feature extraction is performed on the emotion fluctuation data to output key indicators of emotion changes. By combining a pre-established model iteration framework, the parameters of the personalized emotion model are adjusted according to key indicators of emotion changes to generate an updated model version, and the performance of the model is evaluated to see if it has improved. If the model performance improves, then update the personalized emotion model.
8. The multimodal emotion micro-fluctuation tracking method for antidepressant robots according to claim 7, characterized in that, The method further includes: Construct a multi-scenario intervention strategy mapping table; the mapping table contains the correspondence between different environment categories and appropriate intervention methods; Identify the current environment type described by the user; Prioritize the intervention method based on the environment type.
9. A multimodal emotion micro-fluctuation tracking device for antidepressant robots, characterized in that, include: The first processing module is used to: collect facial expression images, voice tone fragments, and physiological indicator monitoring data of patients with depression, and construct an initial multi-source dataset; The second processing module is used to: extract and analyze the facial expression image to obtain preliminary emotional fluctuation feature values by analyzing the frequency and amplitude of micro-expression changes. The third processing module is used to: combine the preliminary emotional fluctuation feature values and voice tone segments to generate an emotional fluctuation index value; The fourth processing module is used to: identify dynamic patterns of emotional change trends based on the emotional fluctuation index values and physiological index monitoring data; The fifth processing module is used to: adjust the parameters of the basic emotion model based on the dynamic pattern of the emotion change trend to obtain a personalized emotion model. Emotional state assessment results are obtained based on personalized emotion models and multi-source data in multi-source datasets. The sixth processing module is used to generate an emotion regulation suggestion signal if the emotion fluctuation index value in the emotion state assessment result exceeds the preset index value. The seventh processing module is used to control the smart device to push customized intervention content based on the emotion regulation suggestion signal.