Human-computer interaction method and device based on virtual reality

Abstract

The invention provides a human-computer interaction method and device based on virtual reality, and the method comprises the steps: S1, obtaining the environment information and initial position information of a user through a virtual reality device, S2, receiving a human body action and an interaction action instruction inputted by the user through a handle, a gesture recognition device and an eyeball tracking device; the method comprises the following steps: S1, acquiring human body actions and interaction action instructions input by a user, S2, processing a rendering task of a virtual environment in parallel through a plurality of processing units, S5, predicting user behaviors by using an intelligent algorithm, rendering and caching related scenes in advance, and S6, monitoring heart rate and blinking frequency parameters of the user in real time, and dynamically adjusting the interaction strength and rhythm of the virtual environment according to the state of the user. The multi-modal input mode of the handle, the gesture recognition device and the eyeball tracking device enables the user to interact with the virtual environment more naturally; parts of tasks are dispersed to the local device and the cloud server, low delay and high efficiency are ensured, and the response speed and stability of the system are improved.

Description

Technical Field

[0001] The present invention relates to a human-computer interaction method and device based on virtual reality, and particularly relates to the field of human-computer interaction. Background Art

[0002] With the development of information technology, virtual reality (VR) technology has become an important research direction for human-computer interaction. In order to provide a more realistic and immersive user experience, modern VR systems integrate advanced technologies such as high-resolution displays, low-latency input / output devices, real-time rendering optimization, intelligent behavior prediction, and physiological parameter monitoring.

[0003] Traditional displays have limitations in refresh rate and resolution, resulting in users being prone to dizziness in fast-moving scenarios; although the human-computer interaction method based on virtual reality has broad application prospects in many fields, there are also some obvious defects: In addition, the resolution and refresh rate of the hardware will also affect the user experience, especially in the display of details and fast-moving scenarios, dizziness or discomfort may occur; in order to achieve a high-quality virtual reality experience, the system requires powerful computing resources, especially when rendering complex scenes and processing a large number of user inputs, which places high requirements on the hardware configuration and energy consumption, and is not conducive to the popularization and portability of the system; it is unable to accurately track the movements of the user's head, hands, and other body parts and reflect them in real time in the virtual environment; this requires high-precision sensors and algorithms to ensure accurate tracking and positioning to provide a smooth and natural user experience. Summary of the Invention

[0004] Object of the Invention: One object is to propose a human-computer interaction method based on virtual reality to solve the above existing problems; a further object is to propose a device for implementing the above method.

[0005] Solution: A human-computer interaction method based on virtual reality, comprising the following steps:

[0006] S1. Obtain the user's environmental information and initial position information through virtual reality devices, and present a three-dimensional virtual environment using a head-mounted display device with a resolution of 3840×2160.

[0007] S2. Receive the human motion and interaction action instructions input by the user through a handle, a gesture recognition device, and an eye tracking device.

[0008] S3. According to the human motion and interaction action instructions input by the user, update the objects and scenes in the three-dimensional virtual environment in real time, and adopt edge computing to distribute some rendering and computing tasks to local devices and cloud servers.

[0009] S4. Render the virtual environment tasks in parallel through multiple processing units, and use dynamic resolution adjustment to adjust the rendering resolution of the virtual environment in real time according to the scene complexity and user perspective changes.

[0010] S5. Use intelligent algorithms to predict user behavior and pre-render and cache relevant scenes.

[0011] S6. Real-time monitor the user's heart rate and blink frequency parameters, and dynamically adjust the interaction intensity and rhythm of the virtual environment according to the user's state.

[0012] In a further embodiment, a low-latency sensor with a latency less than 20 milliseconds and a haptic feedback device are integrated in the virtual reality device; the feedback intensity and frequency of the haptic feedback device are dynamically adjusted according to events in the three-dimensional virtual environment, and the feedback frequency range is 20 Hz to 200 Hz; multi-sensory feedback is provided through a handle with a vibration frequency of 60 Hz.

[0013] The pressure of the handle by the finger is represented by a three-dimensional coordinate system, and the attitude information of the handle is presented in the form of a table.

[0014] In a further embodiment, the gesture recognition device first captures an image of the user's hand through a depth camera.

[0015] Perform real-time processing on the captured image to track the position and movement trajectory of the hand and fingers.

[0016] Separate the hand area from the image to exclude background interference.

[0017] Extract useful information from the image, such as global and local features of the gesture.

[0018] Perform preliminary calculation on the gesture features to determine the basic gesture categories.

[0019] Encode the data after preliminary calculation according to the set data transmission format and transmit it to the virtual reality device through the Bluetooth module.

[0020] The virtual reality device further processes the received data and applies the results to the interaction in the virtual environment.

[0021] In a further embodiment, the three-dimensional virtual environment includes, but is not limited to, OpenGL, Vulkan, DirectX 12, scenes, objects, and interfaces; the initial position information is obtained through a base station and a tracker; the brightness adjustment range of the head-mounted device is from 100 nits to 1000 nits; the three-dimensional virtual environment includes two scenes, respectively for presenting data value coordinate and reproducing virtual scene gesture interaction. The objects and scenes in the three-dimensional virtual environment perform real-time physical simulation and collision detection according to the human body movements and interaction action instructions input by the user, and the physical simulation frame rate is not lower than 90 FPS.

[0022] In a further embodiment, the process of capturing human body movements: When using a virtual reality device, the user needs to fix multiple trackers on the limbs and waist. The tracker includes a six-degree-of-freedom inertial measurement unit, which can track the user's body posture in real time and accurately locate its position through a base station; the base station transmits the motion data of the tracker to a computer, and the computer generates the user's human skeleton through an inverse kinematics algorithm, so as to drive a virtual agent to follow the user's movement in the virtual environment.

[0023] The process of capturing interaction actions: The virtual reality device obtains and processes the user's prop selection information and the interaction information between the user and the prop through the sensors in the handle, and then transmits it to the computer through Bluetooth communication; the computer is responsible for parsing and processing all action capture data, generating a human skeleton model, and managing and coordinating the prop interaction logic; the head-mounted device updates the actions of the virtual agent in real time.

[0024] In a further embodiment, the allocation strategy of the edge computing assigns tasks with a latency lower than 50 milliseconds and higher than the set computing requirements to local devices, while assigning tasks with a latency higher than 50 milliseconds and lower than the set computing requirements to cloud servers.

[0025] In a further embodiment, the rendering task is implemented by using multiple GPU or CPU cores, where the computing power of each GPU is not lower than 10 TFLOPS; the dynamic resolution adjustment increases the resolution to 3840×2160 according to the focus area in the user's field of view, while reducing the resolution to 1920×1080 for the edge area of the field of view.

[0026] In a further embodiment, the intelligent algorithm includes a machine learning model based on historical behavior data, which can predict the user's next action by analyzing the user's historical behavior data.

[0027] The intelligent algorithm, based on the target perception model, establishes a secure network environment for human-computer interaction; uses the Gabor wavelet transform method to analyze the amplitude and phase characteristics of the visual target in human-computer interaction; formulates relevant constraint conditions and objective functions using the support vector machine principle to construct a target perception model for the visual target in human-computer interaction; uses the particle swarm algorithm to optimize in the model to obtain the best model parameters; takes the extracted target features as the input vector of the model and transmits them into the perception model, and completes the perception of the visual target in human-computer interaction based on the model output.

[0028] The real-time monitoring of the user includes but is not limited to physiological parameter sensors, heart rate belts, blood oxygen detectors, and eye trackers, and the data update frequency of the sensors is not less than 1 Hz.

[0029] A virtual reality-based human-computer interaction device for implementing a virtual reality-based human-computer interaction method, including an eye tracking device for obtaining user eye movement data, a depth camera for gesture recognition and environmental perception to capture the user's hand movements and the distance information of other objects, an inertial measurement unit for tracking the user's head and body movements, a base station for accurately positioning the head-mounted display device and the handle position, a physiological parameter sensor for monitoring the user's heart rate, respiratory rate, and skin electrical response physiological signals, a curvature acquisition module for detecting the bending degree of the fingers to assist gesture recognition, a Bluetooth module for wireless communication to connect various peripheral devices, a power supply module for providing power support to ensure the normal operation of each component, a main controller for coordinating and managing the operations of all peripheral devices, headphones, a handle, a software platform, and a haptic feedback device.

[0030] Beneficial effects: The present invention proposes a virtual reality-based human-computer interaction method and device. By using a head-mounted display device with a resolution of 3840×2160 and a refresh rate of 144 Hz, it greatly reduces motion blur and latency, improves visual clarity and smoothness, and reduces the user's sense of dizziness; the multi-modal input methods of the handle, gesture recognition device, and eye tracking device enable the user to interact with the virtual environment more naturally; dispersing some tasks to local devices and cloud servers ensures low latency and high performance, and improves the response speed and stability of the system; adjusting the resolution in real time according to the scene complexity and user perspective changes not only ensures high-definition display of key areas but also effectively saves computing resources and energy consumption.

[0031] Using a machine learning model to predict users' behaviors in advance and preload relevant scenarios significantly reduces latency and stuttering; real-time monitoring of heart rate and blink frequency parameters and dynamically adjusting the interaction intensity of the virtual environment according to the user's state ensures the safety and comfort of use; tactile feedback provides timely and accurate physical feedback, enhancing the immersion feeling, and considering the feedback intensity and frequency of different events to meet the user experience requirements in various situations; the three-dimensional virtual environment rendering engine is compatible with multiple graphics libraries, ensuring the flexibility and technological advancement of content creation;

[0032] The six-degree-of-freedom inertial measurement unit (IMU) combined with base station positioning can accurately track the user's body posture and generate a realistic human skeleton model, making the actions of virtual agents more accurately synchronized and natural; through a depth camera and advanced image processing technology, high-precision recognition of hand movements is achieved, improving the reliability of gesture interaction; the target perception model constructs a safe and reliable interaction environment, optimizing the human-computer interaction experience by analyzing visual target features and improving the intelligence level of the system. Brief Description of the Drawings

[0033] Figure 1 It is a schematic flow diagram of the implementation method of the present invention.

[0034] Figure 2 It is a schematic diagram of human motion capture of the present invention.

[0035] Figure 3 It is a schematic diagram of the device module of the present invention. Detailed Embodiment

[0036] The applicant believes that the resolution and refresh rate of the hardware will also affect the user experience. Especially in scenarios of detailed display and fast movement, dizziness or discomfort may occur; to achieve a high-quality virtual reality experience, the system requires powerful computing resources, especially when rendering complex scenes and processing a large amount of user input, which poses high requirements on the hardware configuration and energy consumption and is not conducive to the popularization and portability of the system.

[0037] To solve the existing problems, the present invention provides a human-computer interaction method and device based on virtual reality. The following will further specifically describe this solution through embodiments and with reference to the drawings.

[0038] In this application, we propose a human-computer interaction method and device based on virtual reality. A human-computer interaction method based on virtual reality included therein comprises the following steps:

[0039] S1. Obtain the user's environmental information and initial position information through a virtual reality device, present a three-dimensional virtual environment using a head-mounted display device with a resolution of 3840×2160, and integrate a low-latency sensor with a latency less than 20 milliseconds and a haptic feedback device in the virtual reality device to provide more timely and accurate user feedback; the feedback intensity and frequency of the haptic feedback device are dynamically adjusted according to events in the three-dimensional virtual environment, and the feedback frequency range is 20Hz to 200Hz; provide multi-sensory feedback through a handle with a vibration frequency of 60Hz, which can not only increase the user's immersion, but also stimulate their attention and enhance their active participation level;

[0040] The pressure of the handle by the finger is represented by a three-dimensional coordinate system, and the attitude information of the handle is presented in tabular form.

[0041] S2. Use a display with a refresh rate of 144Hz to reduce the user's dizziness and discomfort in fast-moving scenarios, and receive the human motion and interaction action instructions input by the user through the handle, gesture recognition device, and eye tracking device; the gesture recognition device first captures an image of the user's hand through a depth camera;

[0042] Perform real-time processing on the captured image to track the position and movement trajectory of the hand and fingers;

[0043] Separate the hand area from the image to exclude background interference and improve the recognition accuracy;

[0044] Extract useful information from the image, such as the global features (color, texture, shape) and local features (corner point type and region type) of the gesture. Especially for vision-based gesture recognition, the key lies in extracting the spatio-temporal features of the gesture;

[0045] Perform a preliminary solution on the gesture features to determine the basic gesture categories;

[0046] Encode the data after the preliminary solution according to the set data transmission format and transmit it to the virtual reality device through a Bluetooth module.

[0047] The virtual reality device further processes the received data and applies the results to the interaction in the virtual environment.

[0048] S3. According to the human body movements and interactive action instructions input by the user, the objects and scenes in the three-dimensional virtual environment are updated in real time. Edge computing is adopted to distribute some rendering and computing tasks to local devices and cloud servers, so as to reduce the computing resource requirements of a single device; relieve the load pressure of a single virtual reality device; update the attribute parameters of the objects and the changes of the scenes in the three-dimensional virtual environment according to the actions and instructions input by the user. Edge computing technology transfers some computing and storage functions to a computer closer to the end user through the network, which can reduce the load of the central node and avoid the bandwidth waste problem caused by transmitting a large amount of data streams at the node. Therefore, in this solution, we can transfer some computationally intensive tasks to local devices or the cloud through edge computing to save a lot of unnecessary computing time and improve the overall operation speed and fluency. The three-dimensional virtual environment includes but is not limited to OpenGL, Vulkan, DirectX12, scenes, objects and interfaces. The initial position information is obtained through base stations and trackers. The brightness adjustment range of the head-mounted display device is from 100 nits to 1000 nits. The three-dimensional virtual environment includes two scenes, which are respectively used for the coordinate presentation of data values and the reproduction and interaction of virtual scene gestures. The objects and scenes in the three-dimensional virtual environment perform real-time physical simulation and collision detection according to the human body movements and interactive action instructions input by the user, and the physical simulation frame rate is not lower than 90 FPS.

[0049] The capture process of human body movements: When the user uses the virtual reality device, multiple trackers need to be fixed on the limbs and waist. The tracker contains a six-degree-of-freedom inertial measurement unit, which can track the user's body posture in real time and accurately locate its position through the base station. The base station transmits the motion data of the tracker to the computer, and the computer generates the user's human skeleton through the inverse kinematics algorithm, so as to drive the virtual agent to follow the user's movement in the virtual environment.

[0050] The capture process of interactive actions: The virtual reality device obtains and processes the user's prop selection information and the interaction information between the user and the prop through the sensors in the handle, and then transmits it to the computer through Bluetooth communication. The computer is responsible for parsing and processing all action capture data, generating a human skeleton model, and managing and coordinating the prop interaction logic. The head-mounted display device updates the actions of the virtual agent in real time.

[0051] The allocation strategy of the edge computing assigns tasks with a latency lower than 50 milliseconds and higher than the set computing requirements to local devices, while assigning tasks with a latency higher than 50 milliseconds and lower than the set computing requirements to the cloud server.

[0052] S4. Implement distributed rendering, where the rendering tasks of the virtual environment are processed in parallel by multiple processing units to improve the rendering efficiency. Utilize dynamic resolution adjustment to adjust the rendering resolution of the virtual environment in real time according to the scene complexity and the change of the user's perspective, so as to optimize the system performance and energy consumption. The rendering tasks are achieved by using multiple GPUs or CPU cores, where the computing power of each GPU is not less than 10 TFLOPS.

[0053] The dynamic resolution adjustment increases the resolution to 3840×2160 according to the focus area in the user's perspective, while reducing the resolution to 1920×1080 for the peripheral area of the field of view.

[0054] The intelligent algorithm includes a machine learning model based on historical behavior data, which can predict the user's next action by analyzing the user's historical behavior data, and the prediction accuracy rate reaches more than 90%.

[0055] S5. Utilize the intelligent algorithm to predict the user's behavior, and pre-render and cache relevant scenes to avoid lags or delays caused by excessive computational burden. At the same time, it can also recommend appropriate content or challenge difficulty according to the user's behavior pattern and learning progress to help them achieve the best learning effect in the virtual world. The intelligent algorithm is based on the target perception model: establish a secure network environment during human-computer interaction; use the Gabor wavelet transform method to analyze the amplitude and phase characteristics of the visual target of human-computer interaction; use the support vector machine principle to formulate relevant constraint conditions and objective functions to construct the target perception model of the visual target of human-computer interaction; adopt the particle swarm algorithm to optimize in the model to obtain the best model parameters; use the extracted target features as the input vector of the model and transmit them into the perception model, and complete the perception of the visual target of human-computer interaction based on the model output.

[0056] S6. Real-time monitor the user's heart rate and blink frequency parameters, and dynamically adjust the interaction intensity and rhythm of the virtual environment according to the user's state. Real-time monitor the user's biological signals, such as heartbeat and breathing. These data can help us understand the user's state, and dynamically adjust the interaction intensity and rhythm of the virtual environment according to their state, which can provide better guidance and suggestions in sports and fitness, making it easier for users to achieve the expected training goals. In addition, it can also be used as a means of psychological intervention to change some bad habits or psychological states of users through the adjustment of the user's body, playing a certain preventive role.

[0057] The real-time monitoring users include, but are not limited to, physiological parameter sensors, heart rate belts, blood oxygen detectors, and eye trackers. The data update frequency of the sensors is not less than 1 Hz. They can be used to monitor users' emotions and physiological characteristics in real time, which helps us better understand users' needs and preferences and provides a reference basis for subsequent design and development of personalized functions. In addition, we can also conduct game interactions through the way of talking to virtual characters with a voice assistant. This form not only increases the fun and attractiveness of the game but also improves the communication and cooperation level among players.

[0058] A virtual reality-based human-computer interaction device for implementing the virtual reality-based human-computer interaction method includes an eye tracking device for acquiring users' eye movement data, a depth camera for gesture recognition and environmental perception to capture users' hand movements and distance information of other objects, an inertial measurement unit for tracking users' head and body movements, a base station for accurately positioning the head-mounted display device and the handle position, a physiological parameter sensor for monitoring users' heart rate, respiratory rate, skin conductance response physiological signals, a curvature acquisition module for detecting the bending degree of fingers to assist gesture recognition, a Bluetooth module for wireless communication to connect various peripheral devices (such as handles, headphones, sensors, etc.), a power supply module for providing power support to ensure the normal operation of each component, a main controller for coordinating and managing the operations of all peripheral devices, headphones, handles, a software platform, and a haptic feedback device. The weight of the head-mounted display device does not exceed 500 grams.

[0059] Through the head-mounted display device, the handle, the sensor, and the software platform, users can enter a completely virtual three-dimensional space and interact with objects in the virtual environment. Users transmit their actions and intentions to the virtual reality system through the handle, gesture recognition, and eye tracking input devices. For example, the handle can be used to simulate grasping and moving virtual objects, and gesture recognition can capture users' hand movements through the camera and convert them into instructions that the system can understand. The system uses a high-performance graphics processor to render a high-quality three-dimensional virtual environment in real time and display it in the head-mounted display device. The rendering engine updates the display content in real time according to the changes in users' positions and perspectives, giving users a sense of immersion. For the haptic feedback device, to enhance users' immersion, the system provides various forms of feedback, such as visual, auditory, and haptic feedback; visual feedback is presented through the display device, auditory feedback plays environmental sound effects or voice prompts through the headphones, and haptic feedback can provide physical feedback through the vibrating handle device.

[0060] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as a limitation of the present invention itself. Various changes can be made in its form and details without departing from the spirit and scope of the present invention defined by the appended claims.

Claims

1. A human-computer interaction method based on virtual reality, characterized in that: Including the following steps: S1. Obtain the user's environment information and initial position information through the virtual reality device, and use the head display device with a resolution of 3840×2160 to present the three-dimensional virtual environment. S2, receiving human body motion and interactive motion instructions input by the user through the handle, gesture recognition device and eye tracking device; S3, based on the human body movements and interactive action instructions input by the user, update the objects and scenes in the 3D virtual environment in real time, and use edge computing to distribute some rendering and computing tasks to local devices and cloud servers. S4, using multiple processing units to process the rendering tasks of the virtual environment in parallel, using dynamic resolution adjustment, and adjusting the rendering resolution of the virtual environment in real time according to the complexity of the scene and the changes in the user's perspective, S5. Use intelligent algorithms to predict user behavior, render and cache related scenes in advance, S6. Monitor the user's heart rate and blinking frequency parameters in real time, and dynamically adjust the interaction intensity and rhythm of the virtual environment according to the user's status.

2. A human-computer interaction method based on virtual reality according to claim 1, characterized in that: A low-latency sensor and a tactile feedback device with a delay of less than 20 milliseconds are integrated into a virtual reality device; the feedback intensity and frequency of the tactile feedback device are dynamically adjusted according to events in a three-dimensional virtual environment, and the feedback frequency range is 20Hz to 200Hz; multi-sensory feedback is provided through a handle with a vibration frequency of 60Hz, The pressure exerted on the handle by the finger is represented by a three-dimensional coordinate system, and the posture information of the handle is presented in a table.

3. The human-computer interaction method based on virtual reality according to claim 1, characterized in that: The gesture recognition device first captures an image of the user's hand through a depth camera; Process the captured images in real time to track the position and movement of hands and fingers; Separate the hand area from the image and exclude background interference. Extract useful information from images, such as global and local features of gestures Perform a preliminary calculation of gesture features to determine basic gesture categories; Encode the initially solved data according to the set data transmission format and transmit it to the virtual reality device via the Bluetooth module; The VR device further processes the received data and applies the results to the interaction in the virtual environment.

4. The human-computer interaction method based on virtual reality according to claim 1, characterized in that: The three-dimensional virtual environment includes but is not limited to OpenGL, Vulkan, DirectX12, scenes, objects and interfaces; the initial position information is obtained through a base station and a tracker; the brightness adjustment range of the head display device is 100 nits to 1000 nits; the three-dimensional virtual environment includes two scenes, which are used for data value coordinate presentation and virtual scene gesture reproduction interaction respectively. The objects and scenes in the three-dimensional virtual environment perform real-time physical simulation and collision detection according to human body movements and interactive action instructions input by the user, and the physical simulation frame rate is not less than 90FPS.

5. The human-computer interaction method based on virtual reality according to claim 1, characterized in that: Human motion capture process: When using virtual reality devices, users need to fix multiple trackers on their limbs and waist. The trackers contain six-degree-of-freedom inertial measurement units, which track the user's body posture in real time and accurately locate their position through base stations. The base stations transmit the tracker's motion data to the computer, which generates the user's human skeleton through inverse kinematics algorithms, so as to drive the virtual agent to follow the user's movements in the virtual environment. The process of capturing interactive actions: The virtual reality device obtains and processes the user's prop selection information and the interaction information between the user and the props through the sensors in the handle, and then transmits it to the computer via Bluetooth communication; the computer is responsible for parsing and processing all motion capture data, generating a human skeleton model, and managing and coordinating prop interaction logic; the head-mounted display device updates the virtual agent's actions in real time.

6. The human-computer interaction method based on virtual reality according to claim 1, characterized in that: The edge computing allocation strategy allocates tasks with a latency lower than 50 milliseconds and higher than the set computing requirements to local devices, and allocates tasks with a latency higher than 50 milliseconds and lower than the set computing requirements to cloud servers.

7. The human-computer interaction method based on virtual reality according to claim 1, characterized in that: The rendering task is implemented by utilizing multiple GPUs or CPU cores, wherein the computing power of each GPU is not less than 10 TFLOPS; The dynamic resolution adjustment increases the resolution to 3840×2160 according to the focus area in the user's field of view, and reduces the resolution to 1920×1080 for the edge area of ​​the field of view.

8. The human-computer interaction method based on virtual reality according to claim 1, characterized in that: The intelligent algorithm includes a machine learning model based on historical behavior data, which can predict the user's next action by analyzing the user's historical behavior data; According to the target perception model, the intelligent algorithm: establishes a safe network environment for human-computer interaction; uses the Gabor wavelet transform method to analyze the amplitude and phase characteristics of the human-computer interaction visual target; uses the support vector machine principle to formulate relevant constraints and objective functions, and constructs a target perception model for the human-computer interaction visual target; uses the particle swarm algorithm to find the best in the model and obtain the best model parameters; uses the extracted target features as the model input vector, transmits them into the perception model, and completes the perception of the human-computer interaction visual target based on the model output; The real-time monitoring of users includes but is not limited to physiological parameter sensors, heart rate belts, blood oxygen detectors and eye trackers, and the data update frequency of the sensors is not less than 1 Hz.

9. A human-computer interaction device based on virtual reality, used to implement a human-computer interaction method based on virtual reality as described in any one of claims 1 to 9, characterized in that: It includes an eye tracking device for obtaining user eye movement data, a depth camera for gesture recognition and environmental perception, capturing user hand movements and other object distance information, an inertial measurement unit for tracking user head and body movements, a base station for accurately positioning the head display device and handle position, a physiological parameter sensor for monitoring the user's heart rate, breathing rate, and skin electrical response physiological signals, a curvature acquisition module for detecting the degree of finger bending and assisting gesture recognition, a Bluetooth module for wireless communication and connecting various peripheral devices, a power supply module that provides power support to ensure the normal operation of each component, a main controller for coordinating and managing the operation of all peripheral devices, headphones, handles, software platforms and tactile feedback devices.

Images