Wearable immersive multi-modal feedback man-machine interaction system and method

Abstract

The invention discloses a wearable immersive multi-modal feedback man-machine interaction system and method, and relates to the technical field of man-machine interaction, the system comprises a sensing module, a physiological monitoring and health management module, a processing module, a feedback module and a user interface module, multiple sensing devices are integrated on a flexible material to form a multi-modal sensor; according to the wearable immersive multi-modal feedback man-machine interaction system and method, by arranging the system and integrating various sensing modules, physiological monitoring and health management, processing and recognition modules and rich feedback mechanisms, highly intelligent and personalized man-machine interaction is achieved through the system; the system can accurately capture the intention and behavior of a user, can monitor the health state in real time and provide timely feedback, and guarantees the instantaneity and high efficiency of data processing in combination with edge calculation and cloud processing. And due to an omnibearing and immersive feedback mechanism and a dynamic user interface, the user experience is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a wearable immersive multimodal feedback human-computer interaction system and method. Background Art

[0002] With the rapid development of science and technology, human-computer interaction (HCI) technology is gradually developing in the direction of immersion and multimodality. The immersive multimodal feedback human-computer interaction system aims to provide users with a more natural, intuitive and highly immersive experience by combining multiple sensory feedback such as touch, vision, sound, and smell. Tactile sensor technology continues to improve its accuracy and sensitivity, and shows broad prospects in multiple application fields; computer vision technology plays an important role in medical, security, intelligent manufacturing and other fields, and the market size continues to expand; acoustic technology has achieved innovations in electroacoustics, communication acoustics and other fields, providing new possibilities for service robots, smart homes, etc.; and digital olfactory technology, as an emerging field, has also shown great potential in food safety, environmental protection, medical health and other aspects, and the market size is expected to continue to grow. The continuous innovation of these technologies and the continuous expansion of application fields will bring a richer, more natural and immersive user experience to the immersive multimodal feedback human-computer interaction system.

[0003] Although immersive multimodal feedback human-computer interaction systems have shown great development potential and prospects, the field is still facing a series of challenges and problems. Among them, technical bottlenecks and complexity are important issues. Since multimodal interaction involves multiple sensory channels and technical fields, it is still difficult to effectively integrate and apply them to actual scenarios. In addition, inconsistent user experience is also an urgent problem to be solved. The technical differences between different devices and platforms may cause users to experience inconsistent experiences when using cross-platforms. Finally, interaction conflicts and misunderstandings are also issues that need attention. In a multimodal interaction environment, interactions between different sensory channels may cause conflicts or misunderstandings, affecting the user experience and the accuracy of system response. Future human-computer interaction systems will pay more attention to multimodal fusion, real-time feedback and active control, and highly immersive experience. An immersive multimodal feedback human-computer interaction system that can integrate multiple sensory channels such as vision, hearing, and touch and provide instant feedback and active control will become the mainstream trend. To this end, we propose a wearable immersive multimodal feedback human-computer interaction system and method. Summary of the invention

[0004] The purpose of the present invention is to provide a wearable immersive multimodal feedback human-computer interaction system and method to solve the problems raised in the above background technology.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solutions: a wearable immersive multimodal feedback human-computer interaction system, the system comprising a perception module, a physiological monitoring and health management module, a processing module, a feedback module and a user interface module;

[0006] The perception module integrates multiple sensing devices on flexible materials to form a multimodal sensor, including voice recognition, facial expression recognition, eye tracking, gesture recognition, tactile feedback and brain wave sensors, and forms a multimodal sensor array to capture the user's interactive intentions and behaviors in real time;

[0007] Physiological monitoring and health management module, with integrated physiological monitoring sensors, monitors the user's physiological health indicators in real time and transmits the detected data to the processing and recognition module in real time;

[0008] The processing and recognition module receives data from the perception module, combines cloud-edge coordinated computing technology, performs preprocessing, feature extraction, and intent recognition in the flexible wearable device, and transmits the processing results to the feedback module and user interface module in real time;

[0009] The feedback module provides multimodal feedback, including visual feedback, auditory feedback, tactile feedback, olfactory feedback and physiological monitoring feedback, based on the output of the processing module and the physiological monitoring and health management module, to enhance the realism and immersion of the interaction;

[0010] The user interface module is responsible for interacting with the user, displaying operation results and information, and receiving further instructions or input from the user.

[0011] Preferably, the multimodal sensor array is distributed at key locations of the system and is used to collect multimodal data of the user's voice, gestures, and expressions.

[0012] Preferably, an edge computing module is added between the perception module and the processing and recognition module, and the edge computing module is directly connected to the multimodal sensor array to quickly receive and process the multimodal sensor data.

[0013] Preferably, the processing and identification module includes a processing unit, which uses a large-model deep learning model to perform feature extraction and intent identification on user data.

[0014] Preferably, the feedback module includes a user interface and an interaction control unit, which generates a feedback signal according to the processing result of the processing and recognition module, and transmits it to the user interface module through a tactile feedback device, and the user interface module realizes interaction with the user.

[0015] Preferably, the multimodal sensor array is connected to the edge computing module via a wired or wireless manner, and the edge computing module is connected to the processing unit via a high-speed interface.

[0016] Preferably, the user module includes a virtual reality display device, and the virtual reality device generates a dynamic interactive scene in combination with user behavior and environmental changes.

[0017] Preferably, the processing unit supports online learning and optimization, and continuously optimizes the system's contextual awareness and adaptive capabilities through machine learning algorithms. As the user interacts with the system in depth, more information and preferences about the user can be gradually learned.

[0018] Preferably, the processing unit communicates with the user interface and the interaction control unit, and the processing unit communicates with the virtual reality display device.

[0019] A method for using a wearable immersive multimodal feedback human-computer interaction system comprises the following steps:

[0020] S1: Initialize the system and start the perception module, processing module, feedback module and user interface module;

[0021] S2: Capture the user's interaction intentions and behaviors in real time through the perception module, including voice, facial expressions, eye movements, gestures, brain waves and other data;

[0022] S3: The processing module pre-processes and extracts features from the captured data, and uses a large model for machine learning to perform intent recognition;

[0023] S4: Based on the recognition results, the feedback module generates corresponding multimodal feedback signals, including visual, auditory, tactile, and olfactory feedback;

[0024] S5: The user interface module displays the operation results and information, and adjusts the interaction process according to user feedback;

[0025] S6: Execute S2 to S5 in a loop to realize a continuous human-computer interaction process.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The invention sets up the system, integrates multiple perception modules, physiological monitoring and health management, processing and recognition modules, and rich feedback mechanisms, and realizes highly intelligent and personalized human-computer interaction. It can not only accurately capture the user's intentions and behaviors, but also monitor the health status in real time and provide timely feedback. It combines edge computing and cloud processing to ensure the immediacy and efficiency of data processing. The all-round, immersive feedback mechanism and dynamic user interface greatly enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0029] Figure 2 It is a schematic diagram of the steps of using the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Example: See Figure 1-Figure 2 ,The present invention provides a technical solution: a wearable immersive multimodal feedback human-computer interaction system, the system includes a perception module, a physiological monitoring and health management module, a processing module, a feedback module and a user interface module;

[0032] The perception module integrates multiple sensing devices on flexible materials to form a multimodal sensor, including voice recognition, facial expression recognition, eye tracking, gesture recognition, tactile feedback and brain wave sensors, and forms a multimodal sensor array to capture the user's interactive intentions and behaviors in real time;

[0033] Physiological monitoring and health management module, with integrated physiological monitoring sensors, monitors the user's physiological health indicators in real time and transmits the detected data to the processing and recognition module in real time;

[0034] The processing and recognition module receives data from the perception module, combines cloud-edge coordinated computing technology, performs preprocessing, feature extraction, and intent recognition in the flexible wearable device, and transmits the processing results to the feedback module and user interface module in real time;

[0035] The feedback module provides multimodal feedback, including visual feedback, auditory feedback, tactile feedback, olfactory feedback and physiological monitoring feedback, based on the output of the processing module and the physiological monitoring and health management module, to enhance the realism and immersion of the interaction;

[0036] The user interface module is responsible for interacting with the user, displaying operation results and information, and receiving further instructions or input from the user.

[0037] As a further limitation of the present invention, a multimodal sensor array is distributed at key positions of the system and is used to collect multimodal data of the user's voice, gestures, and expressions. The multimodal sensor array comprehensively captures the user's voice, gestures, expressions and other data, enhances the naturalness of interaction, improves situational awareness, promotes personalized services, and improves system robustness.

[0038] An edge computing module is added between the perception module and the processing and recognition module. The edge computing module is directly connected to the multimodal sensor array to quickly receive and process multimodal sensor data. An edge computing module is added between the perception module and the processing and recognition module. The module is directly connected to the multimodal sensor array to quickly receive and process massive sensor data, effectively reducing data transmission delays, improving processing efficiency and real-time performance, and bringing users a smoother and more responsive interactive experience.

[0039] The processing and recognition module includes a processing unit, which uses a large-model deep learning model to extract features and recognize intent from user data. The core of the processing and recognition module is the processing unit, which uses large-model deep learning technology to efficiently extract features and accurately recognize intent from user multimodal data, further improving the intelligence level of the system and ensuring a deep understanding of user needs and immediate response;

[0040] The feedback module includes a user interface and interaction control unit. The user interface and interaction control unit generates a feedback signal according to the processing result of the processing and identification module, and transmits it to the user interface module through a tactile feedback device. The user interface module realizes the interaction with the user. The user interface and interaction control unit generates feedback signals in real time according to the in-depth analysis result of the processing and identification module. These signals are seamlessly connected to the user interface module through a tactile feedback device and other media, so as to realize an intuitive and interactive experience with the user.

[0041] The multimodal sensor array is connected to the edge computing module via wired or wireless means, and the edge computing module is connected to the processing unit via a high-speed interface. The multimodal sensor array is tightly integrated with the edge computing module with the help of wired or wireless communication technology to achieve real-time data collection and transmission. The edge computing module relies on the high-speed interface to efficiently connect with the processing unit to ensure rapid data processing and real-time response. This design optimizes the system architecture, improves data transmission efficiency and processing speed, and brings users a smoother and smarter interactive experience.

[0042] The user module includes a virtual reality display device, which generates dynamic interactive scenes by combining user behavior and environmental changes. With its advanced technology, the virtual reality display device can closely integrate user behavior changes with the dynamics of the surrounding environment and generate colorful dynamic interactive scenes in real time. This highly personalized immersive experience not only enhances the user's sense of participation and fun, but also makes human-computer interaction more intuitive, natural and efficient.

[0043] The processing unit supports online learning and optimization. It continuously optimizes the system's situational awareness and self-adaptation capabilities through machine learning algorithms. As users interact more deeply with the system, it can gradually learn more about the user's information and preferences. The processing unit has a built-in powerful online learning and optimization mechanism. Relying on advanced machine learning algorithms, it continuously evolves to improve the system's situational awareness and self-adaptation level. As users interact more deeply with the system, the processing unit can gradually accumulate and learn users' personalized information and preferences, thereby providing more intimate and accurate services and feedback.

[0044] The processing unit communicates with the user interface and the interactive control unit, and the processing unit communicates with the virtual reality display device. As the core hub of the system, the processing unit not only maintains close communication with the user interface and the interactive control unit to ensure the accurate receipt of user commands and the instant transmission of feedback signals; at the same time, it also establishes a direct communication link with the virtual reality display device, and is responsible for the generation and adjustment of dynamic interactive scenes to ensure that users can enjoy a highly immersive visual experience.

[0045] The specific implementation of this embodiment is as follows:

[0046] Perception module:

[0047] Gesture recognition: Using depth cameras and machine learning algorithms to recognize user gestures in real time;

[0048] Speech recognition: integrated high-performance speech recognition engine, supporting continuous speech recognition and natural language understanding;

[0049] Facial expression recognition: Analyze the user's emotional state through facial capture technology and affective computing models;

[0050] Eye tracking: Use high-precision eye tracking equipment to track the user's gaze direction and focus;

[0051] Haptic feedback: embed force feedback sensors in interactive devices (such as handles and gloves) to sense the user's force and direction;

[0052] Brainwave extraction: The brainwave sensor is attached to the user's head to read their brainwaves and analyze their thinking activities.

[0053] By integrating multiple sensing devices such as voice recognition, facial expression recognition, eye tracking, gesture recognition, tactile feedback and brain wave sensors, it is possible to capture the user's interaction intentions and behaviors in an all-round and multi-angle manner, achieving a more natural and intuitive human-computer interaction. The use of flexible materials as the substrate not only improves the wearing comfort, but also enables the device to better fit the human body, improves the accuracy and sensitivity of perception, and realizes deep integration and instant response of multiple sensory channels such as vision, hearing, and touch through a multimodal sensor array.

[0054] Physiological monitoring and health management module:

[0055] Physiological monitoring sensors include heart rate monitors, blood pressure monitors, blood oximeters, body temperature sensors, etc., which monitor the user's physiological health indicators in real time. Based on these data, they can provide health advice, early warning reminders, or automatic assistance in emergencies, turning wearable devices into a powerful assistant for personal health management.

[0056] Processing and recognition module:

[0057] Combining the powerful computing power of the cloud and the low latency characteristics of edge computing, it can achieve rapid data preprocessing, feature extraction and intent recognition, and use deep learning models to conduct in-depth analysis of user data to improve the accuracy and efficiency of intent recognition. It supports online learning and optimization, and continuously improves the system's situational awareness and adaptability through machine learning algorithms. The system's situational awareness and adaptability are continuously optimized through machine learning algorithms. With the deepening of user-system interaction, more information and preferences about the user can be gradually learned. The processing unit uses deep learning algorithms to conduct comprehensive analysis of the user's multimodal data to achieve more accurate understanding of user intent. The combination of edge computing and processing units can reduce data processing delays and improve the level of system intelligence.

[0058] Feedback module:

[0059] Visual feedback: Use VR / AR helmets or high-definition displays to provide an immersive visual experience;

[0060] Auditory feedback: Generate 3D sound effects according to scene needs to enhance the sense of space and direction;

[0061] Haptic feedback: simulate real touch through vibration motors, force feedback gloves and other devices;

[0062] Olfactory feedback: Release specific smells according to the scene needs to enhance immersion;

[0063] It includes vision, hearing, touch, smell and feedback based on physiological monitoring, providing users with a full-range, immersive interactive experience. The user interface and interaction control unit generate feedback signals based on the processing results and transmit them to the user through tactile feedback devices.

[0064] User Interface Module:

[0065] Combine user behavior and environmental changes to generate dynamic interactive scenes to enhance immersion, while receiving further instructions or input from the user to achieve two-way interaction with the user. Virtual reality display devices combine user behavior and environmental changes to generate dynamic interactive scenes to provide an immersive feeling.

[0066] A method for using a wearable immersive multimodal feedback human-computer interaction system comprises the following steps:

[0067] S1: Initialize the system and start the perception module, processing module, feedback module and user interface module;

[0068] S2: Capture the user's interaction intentions and behaviors in real time through the perception module, including voice, facial expressions, eye movements, gestures, brain waves and other data;

[0069] S3: The processing module pre-processes and extracts features from the captured data, and uses a large model for machine learning to perform intent recognition;

[0070] S4: Based on the recognition results, the feedback module generates corresponding multimodal feedback signals, including visual, auditory, tactile, and olfactory feedback;

[0071] S5: The user interface module displays the operation results and information, and adjusts the interaction process according to user feedback;

[0072] S6: Execute S2 to S5 in a loop to realize a continuous human-computer interaction process.

[0073] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A wearable immersive multimodal feedback human-computer interaction system, characterized in that: The system includes a perception module, a physiological monitoring and health management module, a processing module, a feedback module and a user interface module; The perception module integrates multiple sensing devices on flexible materials to form a multimodal sensor, including voice recognition, facial expression recognition, eye tracking, gesture recognition, tactile feedback and brain wave sensors, and forms a multimodal sensor array to capture the user's interactive intentions and behaviors in real time; Physiological monitoring and health management module, with integrated physiological monitoring sensors, monitors the user's physiological health indicators in real time and transmits the detected data to the processing and recognition module in real time; The processing and recognition module receives data from the perception module, combines cloud-edge coordinated computing technology, performs preprocessing, feature extraction, and intent recognition in the flexible wearable device, and transmits the processing results to the feedback module and user interface module in real time; The feedback module provides multimodal feedback, including visual feedback, auditory feedback, tactile feedback, olfactory feedback and physiological monitoring feedback, based on the output of the processing module and the physiological monitoring and health management module, to enhance the realism and immersion of the interaction; The user interface module is responsible for interacting with the user, displaying operation results and information, and receiving further instructions or input from the user.

2. A wearable immersive multimodal feedback human-computer interaction system according to claim 1, characterized in that: The multimodal sensor array is distributed at key locations of the system and is used to collect multimodal data of the user's voice, gestures, and expressions.

3. A wearable immersive multimodal feedback human-computer interaction system according to claim 1, characterized in that: An edge computing module is added between the perception module and the processing and recognition module. The edge computing module is directly connected to the multimodal sensor array to quickly receive and process the multimodal sensor data.

4. A wearable immersive multimodal feedback human-computer interaction system according to claim 1, characterized in that: The processing and identification module includes a processing unit, which uses a large-model deep learning model to perform feature extraction and intent identification on user data.

5. The wearable immersive multimodal feedback human-computer interaction system according to claim 1, characterized in that: The feedback module includes a user interface and an interaction control unit, which generates a feedback signal according to the processing result of the processing and recognition module, and transmits it to the user interface module through a tactile feedback device, and the user interface module realizes interaction with the user.

6. A wearable immersive multimodal feedback human-computer interaction system according to claim 1, characterized in that: The multimodal sensor array is connected to the edge computing module via a wired or wireless manner, and the edge computing module is connected to the processing unit via a high-speed interface.

7. A wearable immersive multimodal feedback human-computer interaction system according to claim 1, characterized in that: The user module includes a virtual reality display device, and the virtual reality device generates a dynamic interactive scene in combination with user behavior and environmental changes.

8. The wearable immersive multi-modal feedback human-computer interaction system according to claim 1, characterized in that: The processing unit supports online learning and optimization, and continuously optimizes the system's situational awareness and adaptive capabilities through machine learning algorithms. As the user interacts with the system in depth, more information and preferences about the user can be gradually learned.

9. The wearable immersive multi-modal feedback human-computer interaction system according to claim 1, characterized in that: The processing unit communicates with the user interface and the interaction control unit, and the processing unit communicates with the virtual reality display device.

10. A method for using a wearable immersive multimodal feedback human-computer interaction system, characterized in that: The following steps are involved: S1: Initialize the system and start the perception module, processing module, feedback module and user interface module; S2: Capture the user's interaction intentions and behaviors in real time through the perception module, including voice, facial expressions, eye movements, gestures, brain waves and other data; S3: The processing module pre-processes and extracts features from the captured data, and uses a large model for machine learning to perform intent recognition; S4: Based on the recognition results, the feedback module generates corresponding multimodal feedback signals, including visual, auditory, tactile, and olfactory feedback; S5: The user interface module displays the operation results and information, and adjusts the interaction process according to user feedback; S6: Execute S2 to S5 in a loop to realize a continuous human-computer interaction process.

Images