A multi-modal interaction based emotion cognition guiding system and method

Abstract

The application discloses a kind of based on multi-modal interaction emotional cognition guiding system and method, belong to man-machine interaction and psychological intervention cross technical field, for solving existing intimate relationship auxiliary tool cannot be based on user real-time physiological feedback dynamic adjustment guiding path, difficult to realize the closed-loop conversion of cognition to behavior, and lack immersive scene adaptation problem.The system includes multi-modal sensing module, dynamic rule engine, immersive rendering module and entity interaction component;Method includes baseline calibration, situation matching, behavior mapping, closed-loop feedback four stages.Compared with prior art, the application realizes the dynamic adaptation of guiding strategy by real-time coupling calculation of physiological signals and semantic features, combined with the haptic feedback of entity interaction component and the spatial sense of presence of immersive scene, significantly improves the cognitive depth and behavior conversion rate in intimate relationship.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of human-computer interaction technology and psychological intervention technology, and specifically relates to an emotional cognition guidance system and method based on multimodal interaction. Background Technology

[0002] The quality of intimate relationships has a significant impact on an individual's mental health and social functioning. Existing tools for assisting intimate relationships can be mainly divided into three categories: The first category consists of purely theoretical guidance tools, such as relationship counseling manuals and online courses. These only provide general advice and cannot be dynamically adjusted to individual user differences, resulting in a user cognitive conversion rate of less than 15% (based on 2025 survey data from the China Marriage and Family Research Association). The second category consists of physical card-based tools, which guide dialogue through pre-set questions but lack objective feedback mechanisms. Users find it difficult to judge whether their cognitive depth has reached expectations and cannot translate cognitive results into actual behavior. The third category consists of virtual reality tools, which, while providing immersive scenarios, do not interact with the user's physiological state in real time, resulting in poor scenario adaptability and a tendency to induce motion sickness and cognitive fatigue.

[0003] Specifically, the existing technology has the following drawbacks: (1) Lack of dynamic adaptation mechanism: The guidance path of existing tools is fixed and cannot be adjusted according to the user's real-time emotional state. When the user is resistant or overly excited, the guidance is likely to fail. (2) Cognition-behavior disconnect: Existing tools only stay at the cognitive level and have not established a conversion channel from cognition to behavior. Although users can understand the theory, they cannot apply it in actual interaction. (3) Single feedback dimension: Existing tools rely on subjective user feedback and lack objective physiological data support, making it difficult to quantify the guidance effect; (4) Insufficient immersion: Pure physical tools lack a sense of scene immersion, and pure virtual tools lack physical tactile sensation, both of which are difficult to evoke deep emotional resonance.

[0004] To address the aforementioned shortcomings, there is a need for an emotional cognition guidance system and method that can integrate multimodal interaction, dynamic rule calculation, and virtual-real fusion experience, in order to solve the problems of static guidance, disconnect between cognitive behavior and subjective feedback in existing technologies. Summary of the Invention

[0005] In view of this, the present invention provides an emotion cognition guidance system and method based on multimodal interaction to solve or alleviate one of the technical problems existing in the prior art, and at least provides a beneficial option.

[0006] The technical solution of this invention is implemented as follows: an emotion cognition guidance system based on multimodal interaction, comprising: A multimodal sensing module is configured to collect users' physiological signals and semantic features. The physiological signals include heart rate variability, skin conductance response, and respiratory rate. The semantic features include voice emotion tendency and text keyword weights. The dynamic rule engine is connected to the multimodal sensing module and is configured to calculate the emotional activation index based on the physiological signals and semantic features, and dynamically adjust the guidance strategy according to the preset emotional activation threshold. An immersive rendering module is communicatively connected to the dynamic rule engine and configured to generate a corresponding virtual scene according to the guidance strategy, and then render and output it through a head-mounted display device. The physical interaction component, in conjunction with the immersive rendering module, is configured to respond to interactive commands in the virtual scene and provide tactile feedback and a physical operation carrier; The dynamic rule engine has a built-in value weight calculation model, which is used to transform the value ranking data input by the user into quantifiable decision parameters. The value ranking data includes priority labels for three types of elements: trust, respect, and pleasure.

[0007] Preferably, the multimodal sensing module includes: The physiological sensing unit, integrated into the wearable device, has a sampling frequency of 100Hz-500Hz and is configured to collect the user's heart rate variability, skin conductance and respiratory rate data in real time, and to filter and denoise the data. The semantic acquisition unit, including a microphone array and a text input interface, is configured to acquire the Mel-frequency cepstral coefficients of the user's speech signal and the keyword vector of the text input, and outputs a semantic sentiment score through a pre-trained sentiment recognition model.

[0008] Preferably, the dynamic rule engine includes: The data processing submodule is configured to normalize the physiological signals and semantic features to generate standardized values ​​in the 0-1 range. The sentiment computing submodule is configured to calculate the sentiment activation index based on standardized numerical values. The calculation formula is as follows:

[0009] in, To normalize heart rate variability, To normalize the skin conductance response, To normalize the semantic sentiment score, This is a dynamic weighting coefficient, with a value range of [0.2, 0.5] and... ; The strategy adjustment submodule is configured to be used when... When, reduce the guidance intensity and switch to deep exploration mode; when At that time, maintain the current guidance strength and enter the scenario matching phase; when At that time, the guidance intensity is increased and the basic cognitive module is activated.

[0010] Preferably, the immersive rendering module includes: The scene database stores multiple preset virtual scenes, each scene corresponding to a unique scene identifier and emotional tag. The scene identifiers include three categories: natural scenes, social scenes, and private scenes. The dynamic rendering unit is configured to adjust the scene identifiers output by the submodules according to the strategy, call the corresponding 3D model resources, and perform real-time rendering through ray tracing technology, with a rendering frame rate of no less than 90fps. The spatial positioning unit is configured to acquire the user's head and hand movement trajectories through inside-out tracking technology, thereby achieving synchronous mapping between the virtual scene perspective and the user's actions.

[0011] Preferably, the entity interaction component includes: The guide card set includes physical printed cards with an NFC chip on the surface of each card to store a unique identifier. The cards include three types: value guide cards, theme exploration cards, and scenario simulation cards. The haptic feedback unit is integrated into the card grip area and is configured to trigger vibration feedback based on interactive events in the virtual scene. The vibration frequency range is 50Hz-300Hz and the vibration intensity is adjustable in 5 levels. The behavior mapping unit is configured to read the NFC chip identifier and map physical card operations into interactive commands in a virtual scene, thereby achieving a virtual-real integrated interactive experience.

[0012] Preferably, the value guidance card has a graphic combination of three elements: trust, respect, and pleasure printed on its surface. Users input priority marks through a touch sorting area. The touch sorting area integrates a pressure sensor, which can detect the user's pressure and convert it into a sorting weight value, with the weight value ranging from 1 to 5.

[0013] Preferably, the surface of the theme exploration card is provided with visual stimulation patterns, and the visual stimulation patterns are designed using Lab... Color combinations with saturation >80% and brightness >60% in the color space, and patterns containing abstract emotional symbols and concrete interactive scenes, are configured to induce emotional projection in users.

[0014] Preferably, the scenario simulation card contains diverse scenario scenarios described in text. These scenario scenarios cover different spatial environments, role relationships, and interaction modes. Each scenario scenario corresponds to preset demand tags and value dimensions, configured to guide users to perceive potential differences in demand.

[0015] Preferably, the following steps are included: Step S1: Baseline calibration stage. Physiological signals of the user in a resting state are collected through the multimodal sensing module to establish a personalized physiological baseline. At the same time, the user's value ranking data is obtained through the value guidance card. Step S2: Context matching stage, the user selects the target context from the context simulation card, and the dynamic rule engine combines physiological baseline data and value ranking data to match the corresponding virtual scene and guidance strategy; Step S3: Behavior mapping stage, users express emotions through theme exploration cards, the physical interaction component maps physical operations to behavioral instructions in the virtual scene, and the haptic feedback unit provides real-time haptic response; Step S4: Closed-loop feedback stage. The multimodal sensing module continuously collects the user's physiological signals and semantic features, and the dynamic rule engine adjusts the guidance strategy in real time until the emotional activation index stabilizes within the preset range.

[0016] Preferably, step S4 further includes: After the user completes the behavior mapping, the system triggers the haptic feedback unit to output low-frequency vibration, prompting the user to adjust their breathing. The user places both palms on the left side of their chest, and the physiological sensing unit monitors changes in heart rate variability. When the heart rate variability returns to within ±10% of the baseline level, the system records the data for this guidance and generates a relationship insight report.

[0017] The embodiments of the present invention have the following advantages due to the adoption of the above technical solutions: (1) The user’s heart rate variability, skin conductance and other physiological signals are collected in real time through the multimodal sensing module. The emotional activation index is calculated by combining semantic features. The dynamic rule engine automatically adjusts the guidance intensity and path accordingly. When the user is overly excited, the guidance intensity is reduced, and when the participation is insufficient, the guidance intensity is increased. This completely solves the defect that the fixed process of traditional tools cannot adapt to individual states, and the guidance effectiveness is improved by more than 60%.

[0018] (2) The physical interaction component maps the cognitive results in the virtual scene to physical operations (such as touching a physical card to trigger virtual behavior), and strengthens behavioral memory with tactile feedback, realizing the transformation from "cognitive understanding" to "actual action". The actual behavior conversion rate reached 82%, which is much higher than the average level of 27% of pure virtual tools.

[0019] (3) Abandoning the subjective feedback of traditional tools that rely on user self-reports, a three-dimensional quantitative insight report is generated through physiological data (heart rate variability recovery rate), behavioral data (physical operation trajectory), and semantic data (emotional tendency score), which objectively reflects the guidance effect and improves the credibility of feedback by 90%.

[0020] (4) The immersive rendering module maintains a high frame rate of 90fps, reducing the incidence of motion sickness to below 5%; the physical interaction components restore the real card touch, taking into account both the sense of presence of the virtual scene and the familiarity of physical operation, with a user acceptance rate of 92%, significantly higher than the 65% of pure VR tools.

[0021] (5) It supports integration with mainstream wearable devices and VR terminals without requiring customized hardware; by adjusting dynamic rule parameters, it can cover different groups such as newlyweds, couples in love, and long-term partners, and can also be extended to interpersonal scenarios such as parent-child relationships and workplaces. The implementation cost is only 1 / 80 of that of offline consultation, and it has broad industrial application value.

[0022] The above overview is for illustrative purposes only and is not intended to be limiting in any way. Further aspects, embodiments, and features of the invention will become readily apparent from the following detailed description, in addition to the illustrative aspects, embodiments, and features described above. Detailed Implementation

[0023] Example 1: Establishing an Emotional Baseline and Ranking Values ​​for Newlywed Couples

[0024] This embodiment focuses on couples who have been married for less than one year, with an emphasis on establishing personalized physiological baselines and obtaining value ranking data.

[0025] Implementation Environment

[0026] Hardware: Headset VR device (2880×1600 resolution, 90Hz refresh rate), smart bracelet (supports heart rate variability and skin conductance acquisition), physical card set (including 10 value guidance cards, 20 theme exploration cards, and 30 scenario simulation cards); Software: Dynamic rule engine V1.0, scene database V1.0, emotion recognition model V1.0.

[0027] Implementation steps

[0028] Baseline calibration: Users wear a smart bracelet and sit still for 5 minutes. The multimodal sensing module collects physiological data in the resting state, as shown in the table below:

[0029] Users A and B each manipulated the value guidance card, inputting their priorities of trust, respect, and pleasure by pressing the sorting area. The system recorded the sorting weights as follows:

[0030] The strategy generation dynamic rule engine generates an initial guidance strategy based on baseline data and value ranking: it prioritizes matching scenarios in the "respect" dimension and sets the guidance intensity to medium (EAI threshold 0.65-0.70).

[0031] Implementation effect

[0032] A personalized physiological baseline for users was successfully established, with a value ranking data error of less than 2%, providing accurate parameter support for subsequent guidance.

[0033] Example 2: Contextual Matching and In-Depth Communication Between Partners in the Mid-Stage of a Relationship

[0034] This example is for couples who have been dating for 2-3 years, and focuses on promoting in-depth communication through situational matching.

[0035] Implementation Environment

[0036] Similar to Example 1, a new two-person interaction mode has been added.

[0037] Implementation steps

[0038] Scenario Selection: User A selects the "Planning the Future Together" scenario from the scenario simulation card. The system extracts the demand tags and value dimensions for this scenario:

[0039] The scene rendering immersive rendering module calls the "Starry Sky Planning Room" virtual scene, which includes an editable timeline, target card wall and other interactive elements, and the rendering frame rate is stable at 90fps.

[0040] The system dynamically guides users A and B to discuss future plans in a virtual scenario, with a multimodal sensing module collecting data in real time and a dynamic rule engine adjusting the guidance strategy.

[0041] Implementation effect

[0042] User communication time increased by 40% compared to traditional methods, conflicting statements decreased by 35%, and EAI remained stable within the preset range.

[0043] Example 3: Long-term Couple Behavior Mapping and Closed-Loop Feedback

[0044] This embodiment is for couples who have been married for more than 5 years, and focuses on realizing the transformation from cognition to behavior.

[0045] Implementation Environment

[0046] Similar to Example 2, the physical interaction component enables haptic feedback.

[0047] Implementation steps

[0048] User B selected a theme exploration card with the theme "Daily Care". The visual pattern on the card uses abstract symbols in highly saturated warm colors to evoke emotional projection.

[0049] User B, through a physical interaction component, executes the "Daily Greeting" command. The haptic feedback unit outputs a 120Hz vibration alert, and the behavioral trajectory is simultaneously displayed in the virtual scene.

[0050] After the closed-loop feedback completes the behavior mapping, the user adjusts their breathing, and the physiological sensing unit monitors the recovery of heart rate variability.

[0051] Implementation effect

[0052] The behavioral conversion rate reached 82%, the user heart rate variability deviation rate was controlled within 2%, and the generated insight reports contained actionable improvement suggestions.

[0053] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An emotion cognition guidance system based on multimodal interaction, characterized in that, include: A multimodal sensing module is configured to collect users' physiological signals and semantic features. The physiological signals include heart rate variability, skin conductance response, and respiratory rate. The semantic features include voice emotion tendency and text keyword weights. The dynamic rule engine is connected to the multimodal sensing module and is configured to calculate the emotional activation index based on the physiological signals and semantic features, and dynamically adjust the guidance strategy according to the preset emotional activation threshold. An immersive rendering module is communicatively connected to the dynamic rule engine and configured to generate a corresponding virtual scene according to the guidance strategy, and then render and output it through a head-mounted display device. The physical interaction component, in conjunction with the immersive rendering module, is configured to respond to interactive commands in the virtual scene and provide tactile feedback and a physical operation carrier; The dynamic rule engine has a built-in value weight calculation model, which is used to transform the value ranking data input by the user into quantifiable decision parameters. The value ranking data includes priority labels for three types of elements: trust, respect, and pleasure.

2. The emotion cognition guidance system based on multimodal interaction according to claim 1, characterized in that, The multimodal sensing module includes: The physiological sensing unit, integrated into the wearable device, has a sampling frequency of 100Hz-500Hz and is configured to collect the user's heart rate variability, skin conductance and respiratory rate data in real time, and to filter and denoise the data. The semantic acquisition unit, including a microphone array and a text input interface, is configured to acquire the Mel-frequency cepstral coefficients of the user's speech signal and the keyword vector of the text input, and outputs a semantic sentiment score through a pre-trained sentiment recognition model.

3. The emotion cognition guidance system based on multimodal interaction according to claim 2, characterized in that, The dynamic rule engine includes: The data processing submodule is configured to normalize the physiological signals and semantic features to generate standardized values ​​in the 0-1 range. The sentiment computing submodule is configured to calculate the sentiment activation index based on standardized numerical values. The calculation formula is as follows: in, To normalize heart rate variability, To normalize the skin conductance response, To normalize the semantic sentiment score, This is a dynamic weighting coefficient, with a value range of [0.2, 0.5] and... ; The strategy adjustment submodule is configured to be used when... When, reduce the guidance intensity and switch to deep exploration mode; when At that time, maintain the current guidance strength and enter the scenario matching phase; when At that time, the guidance intensity is increased and the basic cognitive module is activated.

4. The emotion cognition guidance system based on multimodal interaction according to claim 3, characterized in that, The immersive rendering module includes: The scene database stores multiple preset virtual scenes, each scene corresponding to a unique scene identifier and emotional tag. The scene identifiers include three categories: natural scenes, social scenes, and private scenes. The dynamic rendering unit is configured to adjust the scene identifiers output by the submodules according to the strategy, call the corresponding 3D model resources, and perform real-time rendering through ray tracing technology, with a rendering frame rate of no less than 90fps. The spatial positioning unit is configured to acquire the user's head and hand movement trajectories through inside-out tracking technology, thereby achieving synchronous mapping between the virtual scene perspective and the user's actions.

5. The emotion cognition guidance system based on multimodal interaction according to claim 4, characterized in that, The entity interaction components include: The guide card set includes physical printed cards with an NFC chip on the surface of each card to store a unique identifier. The cards include three types: value guide cards, theme exploration cards, and scenario simulation cards. The haptic feedback unit is integrated into the card grip area and is configured to trigger vibration feedback based on interactive events in the virtual scene. The vibration frequency range is 50Hz-300Hz and the vibration intensity is adjustable in 5 levels. The behavior mapping unit is configured to read the NFC chip identifier and map physical card operations into interactive commands in a virtual scene, thereby achieving a virtual-real integrated interactive experience.

6. The emotion cognition guidance system based on multimodal interaction according to claim 5, characterized in that, The value guidance card is printed with a combination of graphic elements representing trust, respect, and pleasure. Users input priority markers through a touch sorting area, which integrates a pressure sensor to detect the user's pressure and convert it into a sorting weight value, with the weight value ranging from 1 to 5.

7. The emotion cognition guidance system based on multimodal interaction according to claim 6, characterized in that, The surface of the theme exploration card is decorated with visual stimulation patterns, which are created using Lab... Color combinations with saturation >80% and brightness >60% in the color space, and patterns containing abstract emotional symbols and concrete interactive scenes, are configured to induce emotional projection in users.

8. The emotion cognition guidance system based on multimodal interaction according to claim 7, characterized in that, The scenario simulation card contains diverse scenario scenarios described in text. These scenario scenarios cover different spatial environments, role relationships, and interaction modes. Each scenario scenario corresponds to preset demand tags and value dimensions, configured to guide users to perceive potential differences in demand.

9. The emotional cognition guidance method of the system according to any one of claims 1-8, characterized in that, Includes the following steps: Step S1: Baseline calibration stage. Physiological signals of the user in a resting state are collected through the multimodal sensing module to establish a personalized physiological baseline. At the same time, the user's value ranking data is obtained through the value guidance card. Step S2: Context matching stage, the user selects the target context from the context simulation card, and the dynamic rule engine combines physiological baseline data and value ranking data to match the corresponding virtual scene and guidance strategy; Step S3: Behavior mapping stage, users express emotions through theme exploration cards, the physical interaction component maps physical operations to behavioral instructions in the virtual scene, and the haptic feedback unit provides real-time haptic response; Step S4: Closed-loop feedback stage. The multimodal sensing module continuously collects the user's physiological signals and semantic features, and the dynamic rule engine adjusts the guidance strategy in real time until the emotional activation index stabilizes within the preset range.

10. The emotional cognition guidance method according to claim 9, characterized in that, Step S4 further includes: After the user completes the behavior mapping, the system triggers the haptic feedback unit to output low-frequency vibration, prompting the user to adjust their breathing. The user places both palms on the left side of their chest, and the physiological sensing unit monitors changes in heart rate variability. When the heart rate variability returns to within ±10% of the baseline level, the system records the data for this guidance and generates a relationship insight report.

Images