Emotion feedback system based on multiple modes and method thereof

Abstract

The invention relates to the technical field of emotion recognition, in particular to a multi-mode-based emotion feedback system and method, and the method comprises the steps: S1, recognizing feature factors in a current scene; s2, extracting a scene corresponding to the target feedback record, and taking the scene as a reference scene; s3, identifying influence factors contained in the reference scene; and S4, calculating the similarity between the influence factors and the feature factors, screening out target factors in the feature factors according to the similarity, calculating the influence value of the current scene on the emotion of the user, and outputting a recommendation result. By identifying the scene features of the current scene and calculating the influence value of the current scene on the emotion of the user, the possibility of emotion change of the user in the current scene can be predicted in advance, then a prompt can be given, the purpose of predicting the emotion change of the user in advance is achieved, effective measures are taken in time to adjust the emotion of the user, and the user experience is improved. And a user can keep a stable state.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a multimodal-based emotion feedback system and method thereof. Background Art

[0002] The emotional feedback system is based on multimodal technology to achieve emotion recognition and feedback. It uses speech analysis technology to capture changes in tone and speaking speed, facial recognition technology to analyze changes in expression, text analysis technology to process text semantics, and physiological monitoring technology to track changes in heart rate or body temperature. Based on these data, the system uses machine learning and data analysis methods to identify specific emotional patterns and provide personalized feedback or emotion regulation suggestions based on the user's emotional state.

[0003] However, the current emotion feedback system only focuses on analyzing the current instantaneous emotions, that is, analyzing the current user's emotional changes in real time. It cannot predict the user's emotional changes in advance and is not predictive, which leads to the inability to take effective measures in time to regulate the user's emotional changes. Summary of the invention

[0004] The object of the present invention is to provide a multimodal-based emotion feedback system and method thereof to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a multimodal emotional feedback method, the emotional feedback method comprising:

[0006] Step S1: using the smart wearable device to collect images of the user's current scene, construct a three-dimensional model of the current scene, and identify characteristic factors in the current scene;

[0007] Step S2: Obtain the user's emotional feedback records, filter out the target feedback records, extract the scenes corresponding to the target feedback records, and use them as reference scenes;

[0008] Step S3: establishing a recognition model to identify influencing factors contained in the reference scene;

[0009] Step S4: Calculate the similarity between the influencing factors and the characteristic factors, select the target factors from the characteristic factors according to the similarity, calculate the impact value of the current scene on the user's emotions, and output the recommendation results.

[0010] Furthermore, the step S1 includes:

[0011] Step S101: The user wears a smart detection device, which is one or more of a smart bracelet, a smart headset, smart glasses and a smart head-mounted device. The smart detection device is equipped with an image acquisition device, and an image of the user's current environment is captured by the image acquisition device. The image is input into a data model to construct a three-dimensional model, thereby forming a model of the current scene.

[0012] Step S102, identify the things included in the current scene, take one thing as a feature factor, extract all the things included in the current scene, and form a feature set S, where the feature set S = {s1, s2, s3, ..., sn}, where s1, s2, s3, ..., sn represent feature factor 1 included in the current scene, feature factor 2 included in the current scene, feature factor 3 included in the current scene, ..., feature factor n included in the current scene, respectively.

[0013] Furthermore, the step S2 includes:

[0014] Step S201: The user wears the smart wearable device to monitor the user's emotional information in real time, wherein the emotional information includes physiological signal recognition, facial expression recognition, voice recognition, and text emotion analysis recognition;

[0015] The physiological signal recognition process is: monitoring the user's physiological indicators, including heart rate, blood pressure and skin electrophysiological indicators, and judging that the user's emotions have changed when the physiological indicators reach a set transition value;

[0016] The process of facial expression recognition is: collecting the user's facial image and identifying emotions by analyzing facial muscle movements and expression changes.

[0017] The process of speech recognition is: judging emotions through the speech features and intonation changes of the user's speech;

[0018] Text sentiment analysis: infer emotions by analyzing the sentiment words and grammatical structures in the text;

[0019] Step S202: training an emotion recognition model using a training data set, inputting emotion information into the emotion recognition model, and outputting the user's current emotional state, wherein the emotional state includes positive emotions and negative emotions, wherein positive emotions include joy, happiness, satisfaction, happiness, optimism, self-confidence, appreciation, and relaxation; and negative emotions include anxiety, depression, anger, fear, sadness, tension, frustration, and pain;

[0020] Step S203: Each time the emotional state of the user is identified, it is recorded as an emotional feedback record. If the emotional states in two adjacent emotional feedback records are different, they are sorted in order according to the time axis, and the latter of the two adjacent emotional feedback records is marked as the target feedback record;

[0021] Step S204: extract the environment image collected during the recognition process of the target feedback record, input the image into the data model to construct a scene model, and use it as a reference scene.

[0022] Further, step S3 includes:

[0023] Step S301, training a scene recognition model, identifying scene features contained in a reference scene through the scene recognition model, wherein the scene features are things existing in the reference scene, taking each thing in the reference scene as a scene feature, extracting all things existing in the reference scene, and forming a feature set Q, wherein the feature set Q=S={q1, q2, q3, ..., qm}, wherein q1, q2, q3, ..., qm are respectively represented as scene feature 1 contained in the reference scene, scene feature 2 contained in the reference scene, scene feature 3 contained in the reference scene, ..., scene feature m contained in the reference scene;

[0024] Step S302, determining the change type of the target feedback record, wherein the change type includes positive adjustment and negative adjustment, wherein the positive adjustment is when the emotion state identified in the previous emotion feedback record is negative emotion, while the emotion state in the target feedback record is positive emotion; wherein the negative adjustment is when the emotion state identified in the previous emotion feedback record is positive emotion, while the emotion state in the target feedback record is negative emotion;

[0025] Divide the target feedback records according to the change type, extract the target feedback records of positive adjustment into set A, and extract the target feedback records of negative adjustment into set B;

[0026] The scene features corresponding to the target feedback records in set A are formed into set U1, and the scene features corresponding to the target feedback records in set B are formed into set U2. The intersection U3 of U1 and U2 is calculated, and the scene features in sets U1 and U2 that are the same as those in set U3 are eliminated. The remaining scene features in sets U1 and U2 are marked as features to be screened;

[0027] Step S303, obtaining the number of occurrences X of a feature to be screened, and taking the target feedback record corresponding to the feature to be screened as the feature record, recording the number of all target feedback records with the same change type as the feature record as Y, and calculating the influence probability P1 of the feature to be screened, the calculation formula is P1=X / ∑Y;

[0028] The features to be screened whose influence probability P1 is greater than the threshold Pthreshold are marked as influencing factors.

[0029] Further, the step S4 includes:

[0030] Step S401: select a certain characteristic factor and an influencing factor of the same type of thing as the thing included in the characteristic factor, and calculate the similarity between the selected characteristic factor and the influencing factor according to the Euclidean distance. The calculation formula is:

[0031]

[0032] Where d(x, y) represents the straight-line distance between two coordinate points, x represents the score of the object corresponding to the characteristic factor, y represents the score of the object corresponding to the influencing factor, and x represents the score of the object corresponding to the influencing factor. i represents the score of the thing corresponding to the i-th feature factor, y i The score of the thing corresponding to the i-th influencing factor, sim = (x, y) represents the similarity value between the characteristic factor and the influencing factor;

[0033] The characteristic factors with similarity greater than the set threshold are selected as target factors;

[0034] Step S402: When the change type of the target feedback record corresponding to the target factor is positive adjustment, the target factor is marked as a positive factor; when the change type of the target feedback record corresponding to the target factor is negative adjustment, the target factor is marked as a negative factor; wherein the positive factor is 1 and the negative factor is -1.

[0035] Step S403, obtain all target factors included in the current scene, calculate the influence value W of the current scene, and the calculation formula is W=(K1*L1)*β1+(K2*L2)*β1, where K1 is the number of positive factors, L1 represents the positive factor, K2 represents the number of negative factors, L2 represents the negative factor, β1 and β2 are weight factors, which can be obtained through multiple experiments;

[0036] The recommendation results are pushed to the user based on the size of the impact value. When the impact value is larger, it is recommended that the user stay in the current scene longer. When the impact value is smaller, the user is prompted to maintain a stable emotion.

[0037] A multimodal emotion feedback system is applied to a multimodal emotion feedback method, wherein the emotion feedback system comprises a monitoring module, a scene building module, a feature recognition module, a screening module and an output module;

[0038] The monitoring module is used to monitor the user's emotional information and determine whether the user has emotional changes;

[0039] The scene building module uses the smart wearable device to collect images of the user's current scene, build a three-dimensional model of the current scene, and a three-dimensional model of the reference scene, and identify characteristic factors in the current scene and influencing factors in the reference scene;

[0040] The screening module screens out target factors from the characteristic factors according to the similarity between the influencing factors and the characteristic factors;

[0041] The output module is used to calculate the impact value of the current scene on the user's emotions and output the recommendation results;

[0042] The output end of the monitoring module is electrically connected to the input end of the scene building module, the output end of the scene building module is electrically connected to the input ends of the feature recognition module and the screening module, and the output end of the screening module is electrically connected to the input end of the output module.

[0043] Further, the monitoring module includes a user wearing a smart wearable device to monitor the user's emotional information in real time, and the emotional information includes a physiological signal recognition unit, a facial expression recognition unit, a speech recognition unit, a text emotion analysis recognition unit and an emotion recognition model unit;

[0044] The physiological signal recognition unit is used to monitor the user's physiological indicators, including heart rate, blood pressure and skin electrophysiological indicators. When the physiological indicators reach a set transition value, it is determined that the user's emotions have changed;

[0045] The facial expression recognition unit is used to collect the user's facial image and recognize emotions by analyzing facial muscle movements and expression changes;

[0046] The speech recognition unit is used to judge the emotion through the speech characteristics and intonation changes of the user's speech;

[0047] The text sentiment analysis is used to infer emotions by analyzing sentiment words and grammatical structures in the text;

[0048] The emotion recognition model unit is used to input emotion information into the emotion recognition model and output the current emotional state of the user;

[0049] The output ends of the physiological signal recognition unit, the facial expression recognition unit, the speech recognition unit, and the text emotion analysis and recognition unit are electrically connected to the input end of the emotion recognition model unit.

[0050] Furthermore, the screening module includes a screening unit and a calculation unit; the calculation unit is used to calculate the similarity between the selected characteristic factor and the influencing factor according to the Euclidean distance; the screening unit is used to screen the characteristic factors whose similarity is greater than a set threshold as target factors.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention identifies the scene features of the current scene and calculates the impact value of the current scene on the user's emotions, so as to predict in advance the possibility of the user's emotional changes in the current scene, and then issue prompts, thereby achieving the purpose of predicting the user's emotional changes in advance and taking effective measures to regulate the user's emotions in time, so that the user can maintain a stable state. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a flowchart of a multimodal emotional feedback method. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Embodiment 1: Figure 1 As shown, the present invention provides a multimodal emotional feedback method, the emotional feedback method comprising:

[0056] Step S1: using the smart wearable device to collect images of the user's current scene, construct a three-dimensional model of the current scene, and identify characteristic factors in the current scene;

[0057] The step S1 comprises:

[0058] Step S101: The user wears a smart detection device, which is one or more of a smart bracelet, a smart headset, smart glasses and a smart head-mounted device. The smart detection device is equipped with an image acquisition device, and an image of the user's current environment is captured by the image acquisition device. The image is input into a data model to construct a three-dimensional model, thereby forming a model of the current scene.

[0059] Step S102, identify the things included in the current scene, take one thing as a feature factor, extract all the things included in the current scene, and form a feature set S, where the feature set S = {s1, s2, s3, ..., sn}, where s1, s2, s3, ..., sn represent feature factor 1 included in the current scene, feature factor 2 included in the current scene, feature factor 3 included in the current scene, ..., feature factor n included in the current scene, respectively.

[0060] Step S2: Obtain the user's emotional feedback records, filter out the target feedback records, extract the scenes corresponding to the target feedback records, and use them as reference scenes;

[0061] The step S2 comprises:

[0062] Step S201: The user wears the smart wearable device to monitor the user's emotional information in real time, wherein the emotional information includes physiological signal recognition, facial expression recognition, voice recognition, and text emotion analysis recognition;

[0063] The physiological signal recognition process is: monitoring the user's physiological indicators, including heart rate, blood pressure and skin electrophysiological indicators, and judging that the user's emotions have changed when the physiological indicators reach a set transition value;

[0064] The process of facial expression recognition is: collecting the user's facial image and identifying emotions by analyzing facial muscle movements and expression changes.

[0065] The process of speech recognition is: judging emotions through the speech features and intonation changes of the user's speech;

[0066] Text sentiment analysis: infer emotions by analyzing the sentiment words and grammatical structures in the text;

[0067] Step S202: training an emotion recognition model using a training data set, inputting emotion information into the emotion recognition model, and outputting the user's current emotional state, wherein the emotional state includes positive emotions and negative emotions, wherein positive emotions include joy, happiness, satisfaction, happiness, optimism, self-confidence, appreciation, and relaxation; and negative emotions include anxiety, depression, anger, fear, sadness, tension, frustration, and pain;

[0068] Step S203: Each time the emotional state of the user is identified, it is recorded as an emotional feedback record. If the emotional states in two adjacent emotional feedback records are different, they are sorted in order according to the time axis, and the latter of the two adjacent emotional feedback records is marked as the target feedback record;

[0069] Step S204: extract the environment image collected during the recognition process of the target feedback record, input the image into the data model to construct a scene model, and use it as a reference scene.

[0070] Step S3: establishing a recognition model to identify influencing factors contained in the reference scene;

[0071] Step S3 includes:

[0072] Step S301, training a scene recognition model, identifying scene features contained in a reference scene through the scene recognition model, wherein the scene features are things existing in the reference scene, taking each thing in the reference scene as a scene feature, extracting all things existing in the reference scene, and forming a feature set Q, wherein the feature set Q=S={q1, q2, q3, ..., qm}, wherein q1, q2, q3, ..., qm are respectively represented as scene feature 1 contained in the reference scene, scene feature 2 contained in the reference scene, scene feature 3 contained in the reference scene, ..., scene feature m contained in the reference scene;

[0073] Step S302, determining the change type of the target feedback record, wherein the change type includes positive adjustment and negative adjustment, wherein the positive adjustment is when the emotion state identified in the previous emotion feedback record is negative emotion, while the emotion state in the target feedback record is positive emotion; wherein the negative adjustment is when the emotion state identified in the previous emotion feedback record is positive emotion, while the emotion state in the target feedback record is negative emotion;

[0074] Divide the target feedback records according to the change type, extract the target feedback records of positive adjustment into set A, and extract the target feedback records of negative adjustment into set B;

[0075] The scene features corresponding to the target feedback records in set A are formed into set U1, and the scene features corresponding to the target feedback records in set B are formed into set U2. The intersection U3 of U1 and U2 is calculated, and the scene features in sets U1 and U2 that are the same as those in set U3 are eliminated. The remaining scene features in sets U1 and U2 are marked as features to be screened;

[0076] Step S303, obtaining the number of occurrences X of a feature to be screened, and taking the target feedback record corresponding to the feature to be screened as the feature record, recording the number of all target feedback records with the same change type as the feature record as Y, and calculating the influence probability P1 of the feature to be screened, the calculation formula is P1=X / ∑Y;

[0077] The features to be screened whose influence probability P1 is greater than the threshold Pthreshold are marked as influencing factors.

[0078] Step S4: Calculate the similarity between the influencing factors and the characteristic factors, select the target factors from the characteristic factors according to the similarity, calculate the impact value of the current scene on the user's emotions, and output the recommendation results.

[0079] The step S4 comprises:

[0080] Step S401: select a certain characteristic factor and an influencing factor of the same type of thing as the thing included in the characteristic factor, and calculate the similarity between the selected characteristic factor and the influencing factor according to the Euclidean distance. The calculation formula is:

[0081]

[0082] Where d(x, y) represents the straight-line distance between two coordinate points, x represents the score of the object corresponding to the characteristic factor, y represents the score of the object corresponding to the influencing factor, and x represents the score of the object corresponding to the influencing factor. i represents the score of the thing corresponding to the i-th feature factor, y i The score of the thing corresponding to the i-th influencing factor, sim = (x, y) represents the similarity value between the characteristic factor and the influencing factor;

[0083] The characteristic factors with similarity greater than the set threshold are selected as target factors;

[0084] Step S402: When the change type of the target feedback record corresponding to the target factor is positive adjustment, the target factor is marked as a positive factor; when the change type of the target feedback record corresponding to the target factor is negative adjustment, the target factor is marked as a negative factor; wherein the positive factor is 1 and the negative factor is -1.

[0085] Step S403, obtain all target factors included in the current scene, calculate the influence value W of the current scene, and the calculation formula is W=(K1*L1)*β1+(K2*L2)*β1, where K1 is the number of positive factors, L1 represents the positive factor, K2 represents the number of negative factors, L2 represents the negative factor, β1 and β2 are weight factors, which can be obtained through multiple experiments;

[0086] The recommendation results are pushed to the user based on the size of the impact value. When the impact value is larger, it is recommended that the user stay in the current scene longer. When the impact value is smaller, the user is prompted to maintain a stable emotion.

[0087] Embodiment 2: A multimodal emotion feedback system, which is applied to a multimodal emotion feedback method, the emotion feedback system includes a monitoring module, a scene building module, a feature recognition module, a screening module and an output module; the output end of the monitoring module is electrically connected to the input end of the scene building module, the output end of the scene building module is electrically connected to the input ends of the feature recognition module and the screening module, and the output end of the screening module is electrically connected to the input end of the output module.

[0088] The monitoring module is used to monitor the user's emotional information and determine whether the user has emotional changes; the monitoring module includes a user wearing a smart wearable device to monitor the user's emotional information in real time, and the emotional information includes a physiological signal recognition unit, a facial expression recognition unit, a speech recognition unit, a text emotion analysis and recognition unit and an emotion recognition model unit; the output ends of the physiological signal recognition unit, the facial expression recognition unit, the speech recognition unit, and the text emotion analysis and recognition unit are electrically connected to the input end of the emotion recognition model unit.

[0089] The physiological signal recognition unit is used to monitor the user's physiological indicators, which include heart rate, blood pressure and skin electrophysiological indicators. When the physiological indicators reach a set transition value, it is judged that the user's emotions have changed; the facial expression recognition unit is used to collect the user's facial image and identify emotions by analyzing facial muscle movements and expression changes; the speech recognition unit is used to judge emotions through the voice characteristics and intonation changes of the user's speech; the text sentiment analysis is used to infer emotions by analyzing the emotional vocabulary and grammatical structure in the text; the emotion recognition model unit is used to input emotion information into the emotion recognition model and output the user's current emotional state.

[0090] The scene building module uses the smart wearable device to collect images of the user's current scene, build a three-dimensional model of the current scene, and a three-dimensional model of the reference scene, and identify characteristic factors in the current scene and influencing factors in the reference scene;

[0091] The screening module screens out target factors from the characteristic factors according to the similarity between the influencing factors and the characteristic factors; the screening module includes a screening unit and a calculation unit; the calculation unit is used to calculate the similarity between the selected characteristic factors and the influencing factors according to the Euclidean distance; the screening unit is used to screen the characteristic factors whose similarity is greater than a set threshold as target factors.

[0092] The output module is used to calculate the impact value of the current scene on the user's emotions and output the recommendation results.

[0093] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A multimodal emotional feedback method, characterized by: The emotional feedback method comprises: Step S1: using the smart wearable device to collect images of the user's current scene, construct a three-dimensional model of the current scene, and identify characteristic factors in the current scene; Step S2: Obtain the user's emotional feedback records, filter out the target feedback records, extract the scenes corresponding to the target feedback records, and use them as reference scenes; Step S3: establishing a recognition model to identify influencing factors contained in the reference scene; Step S4: Calculate the similarity between the influencing factors and the characteristic factors, select the target factors from the characteristic factors according to the similarity, calculate the impact value of the current scene on the user's emotions, and output the recommendation results.

2. The multimodal emotional feedback method according to claim 1, characterized in that: The step S1 comprises: Step S101: The user wears a smart detection device, which is one or more of a smart bracelet, a smart headset, smart glasses and a smart head-mounted device. The smart detection device is equipped with an image acquisition device, and an image of the user's current environment is captured by the image acquisition device. The image is input into a data model to construct a three-dimensional model, thereby forming a model of the current scene. Step S102, identify the things included in the current scene, take one thing as a feature factor, extract all the things included in the current scene, and form a feature set S, where the feature set S = {s1, s2, s3, ..., sn}, where s1, s2, s3, ..., sn represent feature factor 1 included in the current scene, feature factor 2 included in the current scene, feature factor 3 included in the current scene, ..., feature factor n included in the current scene, respectively.

3. The multimodal emotional feedback method according to claim 1, characterized in that: The step S2 comprises: Step S201: The user wears the smart wearable device to monitor the user's emotional information in real time, wherein the emotional information includes physiological signal recognition, facial expression recognition, voice recognition, and text emotion analysis recognition; The physiological signal recognition process is: monitoring the user's physiological indicators, including heart rate, blood pressure and skin electrophysiological indicators, and judging that the user's emotions have changed when the physiological indicators reach a set transition value; The facial expression recognition process is as follows: collecting the user's facial image and identifying emotions by analyzing facial muscle movements and expression changes; The process of speech recognition is: judging emotions through the speech features and intonation changes of the user's speech; Text sentiment analysis: infer emotions by analyzing the sentiment words and grammatical structures in the text; Step S202: training an emotion recognition model using a training data set, inputting emotion information into the emotion recognition model, and outputting the user's current emotional state, wherein the emotional state includes positive emotions and negative emotions, wherein positive emotions include joy, happiness, satisfaction, happiness, optimism, self-confidence, appreciation, and relaxation; and negative emotions include anxiety, depression, anger, fear, sadness, tension, frustration, and pain; Step S203: Each time the emotional state of the user is identified, it is recorded as an emotional feedback record. If the emotional states in two adjacent emotional feedback records are different, they are sorted in order according to the time axis, and the latter of the two adjacent emotional feedback records is marked as the target feedback record; Step S204: extract the environment image collected during the recognition process of the target feedback record, input the image into the data model to construct a scene model, and use it as a reference scene.

4. The multimodal emotional feedback method according to claim 3, characterized in that: Step S3 includes: Step S301, training a scene recognition model, identifying scene features contained in a reference scene through the scene recognition model, wherein the scene features are things existing in the reference scene, taking each thing in the reference scene as a scene feature, extracting all things existing in the reference scene, and forming a feature set Q, wherein the feature set Q=S={q1, q2, q3, ..., qm}, wherein q1, q2, q3, ..., qm are respectively represented as scene feature 1 contained in the reference scene, scene feature 2 contained in the reference scene, scene feature 3 contained in the reference scene, ..., scene feature m contained in the reference scene; Step S302, determining the change type of the target feedback record, wherein the change type includes positive adjustment and negative adjustment, wherein the positive adjustment is when the emotion state identified in the previous emotion feedback record is negative emotion, while the emotion state in the target feedback record is positive emotion; wherein the negative adjustment is when the emotion state identified in the previous emotion feedback record is positive emotion, while the emotion state in the target feedback record is negative emotion; Divide the target feedback records according to the change type, extract the target feedback records of positive adjustment into set A, and extract the target feedback records of negative adjustment into set B; The scene features corresponding to the target feedback records in set A are formed into set U1, and the scene features corresponding to the target feedback records in set B are formed into set U2. The intersection U3 of U1 and U2 is calculated, and the scene features in sets U1 and U2 that are the same as those in set U3 are eliminated. The remaining scene features in sets U1 and U2 are marked as features to be screened; Step S303, obtaining the number of occurrences X of a feature to be screened, and taking the target feedback record corresponding to the feature to be screened as the feature record, recording the number of all target feedback records with the same change type as the feature record as Y, and calculating the influence probability P1 of the feature to be screened, the calculation formula is P1=X / ∑Y; The features to be screened whose influence probability P1 is greater than the threshold Pthreshold are marked as influencing factors.

5. The multimodal emotional feedback method according to claim 4, characterized in that: The step S4 comprises: Step S401: select a certain characteristic factor and influencing factors of the same type of things as the things included in the characteristic factor, calculate the similarity between the selected characteristic factor and the influencing factors according to the Euclidean distance, and select the characteristic factors with similarity greater than a set threshold as target factors; Step S402: When the change type of the target feedback record corresponding to the target factor is positive adjustment, the target factor is marked as a positive factor; when the change type of the target feedback record corresponding to the target factor is negative adjustment, the target factor is marked as a negative factor; wherein the positive factor is 1 and the negative factor is -1. Step S403: Obtain all target factors included in the current scene, and calculate the influence value W of the current scene. The calculation formula is W=(K1*L1)*β1+(K2*L2)*β1, where K1 is the number of positive factors, L1 represents the positive factor, K2 represents the number of negative factors, L2 represents the negative factor, and β1 and β2 are weight factors; The recommendation results are pushed to the user based on the size of the impact value. When the impact value is larger, it is recommended that the user stay in the current scene longer. When the impact value is smaller, the user is prompted to maintain a stable emotion.

6. A multimodal emotion feedback system, applied to a multimodal emotion feedback method according to any one of claims 1 to 5, characterized in that: The emotional feedback system includes a monitoring module, a scene building module, a feature recognition module, a screening module and an output module; The monitoring module is used to monitor the user's emotional information and determine whether the user has emotional changes; The scene building module uses the smart wearable device to collect images of the user's current scene, build a three-dimensional model of the current scene, and a three-dimensional model of the reference scene, and identify characteristic factors in the current scene and influencing factors in the reference scene; The screening module screens out target factors from the characteristic factors according to the similarity between the influencing factors and the characteristic factors; The output module is used to calculate the impact value of the current scene on the user's emotions and output the recommendation results; The output end of the monitoring module is electrically connected to the input end of the scene building module, the output end of the scene building module is electrically connected to the input ends of the feature recognition module and the screening module, and the output end of the screening module is electrically connected to the input end of the output module.

7. The multimodal emotional feedback system according to claim 6, characterized in that: The monitoring module includes a user wearing a smart wearable device to monitor the user's emotional information in real time, and the emotional information includes a physiological signal recognition unit, a facial expression recognition unit, a speech recognition unit, a text emotion analysis recognition unit and an emotion recognition model unit; The physiological signal recognition unit is used to monitor the user's physiological indicators, including heart rate, blood pressure and skin electrophysiological indicators. When the physiological indicators reach a set transition value, it is determined that the user's emotions have changed; The facial expression recognition unit is used to collect the user's facial image and recognize emotions by analyzing facial muscle movements and expression changes; The speech recognition unit is used to judge the emotion through the speech characteristics and intonation changes of the user's speech; The text sentiment analysis is used to infer emotions by analyzing sentiment words and grammatical structures in the text; The emotion recognition model unit is used to input emotion information into the emotion recognition model and output the current emotional state of the user; The output ends of the physiological signal recognition unit, the facial expression recognition unit, the speech recognition unit, and the text emotion analysis and recognition unit are electrically connected to the input end of the emotion recognition model unit.

8. The multimodal emotional feedback system according to claim 6, characterized in that: The screening module includes a screening unit and a calculation unit; the calculation unit is used to calculate the similarity between the selected characteristic factor and the influencing factor according to the Euclidean distance; the screening unit is used to screen the characteristic factors whose similarity is greater than a set threshold as target factors.

Images