Interaction method with tactile feedback, glove and virtual reality equipment

Abstract

The invention relates to the technical field of VR, provides an interaction method with tactile feedback, gloves and virtual reality equipment, and is used for improving the flexibility and authenticity of interaction. According to the method, a plurality of infrared sensors and a plurality of motion sensors are installed on the hand back side of a glove, a virtual reality device determines the initial pose and orientation of the glove through a glove infrared image collected by a camera, the initial pose serves as prior, 3D gesture recognition is conducted through the glove infrared image, the accuracy and efficiency of gesture recognition are improved, and the user experience is improved. The initial pose is optimized by using motion measurement data acquired by the plurality of motion sensors, so that accurate positioning and 3D gesture recognition of the glove are realized, the accuracy and response speed of interaction between the glove and virtual reality equipment are improved, and meanwhile, a plurality of tactile feedback devices are mounted on the palm side of the glove, so that the accuracy and response speed of interaction between the glove and the virtual reality equipment are improved. The limitation of handle interaction and gesture interaction is solved, a more flexible operation mode and more real tactile feedback can be provided, and the immersive experience of a user is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology, and provides an interaction method, gloves and virtual reality equipment with tactile feedback. Background Art

[0002] At present, virtual reality devices often use handles or gestures as interaction methods. Among them, when based on handle interaction, the user controls the user interface displayed by the virtual reality device and the virtual objects in the virtual screen by holding the handle and operating the buttons. This interaction method is simple to operate, but because the handle needs to be held by the user, the freedom of two-handed operation is limited to a certain extent, and it is not suitable for application scenarios that require free and flexible operation. When the user holds the handle to interact, the vibration motor built into the handle only provides basic force feedback to the palm, lacks richer tactile feedback, and cannot simulate the real feeling of touching the object, resulting in a poor interaction experience. When based on gesture interaction, the user's gesture is captured by using a camera or a depth sensor, and mapped to the virtual environment to operate the user interface and the virtual objects in the virtual screen. Compared with handle interaction, this method can achieve a certain degree of freedom and gesture control, but the accuracy and stability of the current single gesture recognition technology are not high, and there are still certain limitations in the accuracy of complex gesture recognition, especially in terms of stability in fast movements and complex scenes. In addition, since gesture recognition technology mainly relies on visual input, it cannot provide real tactile feedback, thereby reducing the user's sense of immersion.

[0003] With the development of deep learning and chip technology, some gloves that interact with virtual reality devices have appeared on the market. Various sensors are integrated in the gloves, which can provide a more flexible operation method than handle interaction and a more realistic tactile feedback interaction method than gesture interaction. However, existing gloves with tactile feedback have problems such as complex structure, high cost, low sensor accuracy, delayed and inaccurate glove positioning results, which affect the accuracy and response speed of the interaction between the gloves and virtual reality devices and the authenticity of the tactile feedback. Summary of the invention

[0004] The embodiments of the present application provide an interaction method, a glove, and a virtual reality device with tactile feedback, which are used to improve the accuracy and response speed of the interaction between the glove and the virtual reality device while enhancing the authenticity of the user's tactile feedback.

[0005] On the one hand, an embodiment of the present application provides an interaction method with tactile feedback, which is applied to a virtual reality device, wherein the virtual reality device interacts with a headgear, and the glove is equipped with multiple infrared sensors, multiple tactile feedback devices, and multiple motion sensors. The method includes:

[0006] Acquire an infrared image of the glove taken by a camera when the infrared sensor is on, and acquire motion measurement data of the glove collected by the motion sensor;

[0007] Detecting the infrared spot in the infrared image of the glove, and determining the initial position and orientation of the glove in the global coordinate system in combination with the intrinsic parameters of the camera obtained through calibration;

[0008] Recognize the 3D gesture of the glove according to the initial posture and each infrared spot in the infrared image of the glove;

[0009] Optimizing the initial posture according to the motion measurement data to determine the target posture of the glove in a global coordinate system;

[0010] Displaying a glove model interacting with a virtual object in a virtual screen according to the 3D gesture, the target posture, and the orientation;

[0011] According to the attributes of the virtual object, the parameters of the tactile feedback device are adjusted so that the glove provides the target user with a tactile sense of contact between the hand and the virtual object according to the parameters.

[0012] On the other hand, an embodiment of the present application provides a virtual reality device, including a processor, a memory, a display, a camera, and a communication interface, wherein the communication interface, the camera, the display, the memory, and the processor are connected via a bus;

[0013] The communication interface is used to interact with the glove, and the glove is equipped with multiple infrared sensors, multiple tactile feedback devices and multiple motion sensors;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Acquire an infrared image of the glove taken by the camera when the infrared sensor is on, and acquire motion measurement data of the glove collected by the motion sensor;

[0016] Detecting the infrared spot in the infrared image of the glove, and determining the initial position and orientation of the glove in the global coordinate system in combination with the intrinsic parameters of the camera obtained through calibration;

[0017] Recognize the 3D gesture of the glove according to the initial posture and each infrared spot in the infrared image of the glove;

[0018] Optimizing the initial posture according to the motion measurement data to determine the target posture of the glove in a global coordinate system;

[0019] Displaying a glove model interacting with a virtual object in a virtual picture displayed on the display according to the 3D gesture, the target posture and the orientation;

[0020] According to the attributes of the virtual object, the parameters of the tactile feedback device are adjusted so that the glove provides the target user with a tactile sense of contact between the hand and the virtual object according to the parameters.

[0021] Optionally, the processor detects the infrared spot in the infrared image of the glove, and determines the initial position and orientation of the glove in the global coordinate system in combination with the internal and external parameters of the camera obtained by calibration, and the specific operation is:

[0022] Using a feature point detection algorithm to detect the infrared image of the glove, and obtaining the initial 2D coordinates of each infrared spot in the infrared image of the glove;

[0023] According to the position structure of the infrared sensor on the glove, each initial 2D coordinate is optimized to obtain the target 2D coordinate of each infrared spot;

[0024] Obtaining the 3D coordinates of each infrared sensor on the glove, and determining the initial position and posture of the glove in the global coordinate system by combining the intrinsic parameters of the camera and the target 2D coordinates of each infrared spot;

[0025] According to the target 2D coordinates of each infrared light spot and the intrinsic parameters of the camera, the camera coordinates of each infrared light spot in the camera coordinate system are determined, and each camera coordinate is normalized to determine the orientation of the glove.

[0026] Optionally, the processor recognizes the 3D gesture of the glove according to the initial posture and each infrared spot in the infrared image of the glove, and the specific operation is:

[0027] According to the initial posture, obtaining a target area of ​​the hand in the infrared image of the glove;

[0028] Performing feature extraction on the target area to obtain hand state features, and performing hand key point detection on the target area to obtain hand skeleton features;

[0029] Taking each infrared light spot as a reference point, obtaining the hand change characteristics of the glove relative to each reference point;

[0030] According to the hand state feature, the hand skeleton feature and the hand change feature, a gesture label of the glove is determined to obtain a 3D gesture of the glove.

[0031] Optionally, the motion measurement data includes acceleration and angular velocity, and the processor optimizes the initial posture according to the motion measurement data to determine the target posture of the glove in the global coordinate system, and the specific operation is:

[0032] Initialize a state vector, a state covariance matrix, a noise covariance matrix, a measurement noise covariance matrix, a state transfer matrix, and a measurement matrix of a Kalman filter; wherein the state vector is determined based on the integral of the acceleration and the angular velocity;

[0033] Predicting the state vector at the current moment based on the state transfer matrix and the state vector at the previous moment;

[0034] Predicting the state covariance matrix at the current moment according to the noise covariance matrix, the state transfer matrix and the state covariance matrix at the previous moment;

[0035] Calculating a Kalman filter gain according to the state covariance matrix, the measurement matrix and the measurement noise covariance matrix at the current moment;

[0036] The initial posture is optimized according to the state vector at the current moment, the Kalman filter gain and the measurement matrix to obtain a target posture of the glove in a global coordinate system.

[0037] Optionally, the glove is divided into two layers, an inner layer and an outer layer, and the plurality of infrared sensors are installed on the fingertips and joints of each finger on the back of the hand of the outer layer of the glove and on the back of the hand metacarpal bone;

[0038] The multiple tactile feedback devices are installed on the fingertips and joints of each finger on the palm side of the outer layer of the glove and on the palm metacarpal bones;

[0039] The plurality of motion sensors are installed on the wrist of the back of the hand on the inner layer of the glove.

[0040] Optionally, the outer layer of the gloves is made of lightweight stretch fabric, and the inner layer of the gloves is made of sweat-absorbent fabric, and the inner layer of the gloves is removable, washable and replaceable.

[0041] Optionally, the tactile feedback device is a linear resonance motor, and the parameters of the tactile feedback device include at least one of input current amplitude, input current frequency and mechanical characteristics; wherein the input current amplitude and the mechanical characteristics are used to change the vibration intensity of the linear resonance motor; and the input current frequency is used to change the vibration frequency of the linear resonance motor.

[0042] On the other hand, an embodiment of the present application provides a glove for interacting with a virtual reality device, the glove being divided into two layers, an inner layer and an outer layer, a plurality of infrared sensors being installed on the fingertips and joints of each finger on the back of the hand and on the metacarpal bones of the back of the hand on the outer layer of the glove, a plurality of tactile feedback devices being installed on the fingertips and joints of each finger on the palm side of the outer layer of the glove and on the metacarpal bones of the palm, a plurality of motion sensors being installed on the wrist on the back of the hand on the inner layer of the glove, the outer layer of the glove being made of a lightweight stretch fabric, the inner layer of the glove being made of a sweat-absorbent material, and the inner layer of the glove being removable, washable and replaceable.

[0043] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of an interaction method with tactile feedback provided in an embodiment of the present application.

[0044] The beneficial effects of an interactive method with tactile feedback provided by an embodiment of the present application are as follows:

[0045] The gloves that interact with the virtual reality device are equipped with multiple infrared sensors, multiple tactile feedback devices and multiple motion sensors, so as to realize the interactive experience with tactile sense. During the interaction process, the initial position and orientation of the gloves in the global coordinate system are obtained through visual positioning technology according to the infrared spots detected in the infrared image of the gloves taken by the camera and the internal parameters of the camera. In this way, the initial position and infrared spots of the gloves can be used as the priori for gesture recognition of the infrared image of the gloves, improving the efficiency and accuracy of gesture recognition, and then improving the response speed and accuracy of the interaction between the gloves and the virtual reality device; at the same time, the motion measurement data collected by the motion sensor on the gloves is used to estimate the glove position, and the initial position of the infrared sensor positioning is optimized according to the estimation result, which effectively improves the positioning accuracy of the gloves. Furthermore, according to the optimized target position, the recognized 3D gesture and the orientation of the gloves, the glove model that interacts with the virtual object can be displayed in the virtual screen, and the tactile sense of the target user's hand interacting with the virtual object is given through the tactile feedback device on the gloves, providing the target user with an immersive experience like being in the scene.

[0046] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1A This is a schematic diagram of existing gloves;

[0049] Figure 1B This is a schematic diagram of existing gloves;

[0050] Figure 1C This is a schematic diagram of existing gloves;

[0051] Figure 1D This is a schematic diagram of existing gloves;

[0052] Figure 1E This is a schematic diagram of existing gloves;

[0053] Figure 1F This is a schematic diagram of existing gloves;

[0054] Figure 2A A schematic diagram of a glove with an infrared sensor provided in an embodiment of the present application;

[0055] Figure 2B A schematic diagram of a glove with a tactile sensor provided in an embodiment of the present application;

[0056] Figure 2C A side view of a glove provided in an embodiment of the present application;

[0057] Figure 3 An interactive architecture diagram with tactile feedback provided in an embodiment of the present application;

[0058] Figure 4 A flowchart of an interactive method with tactile feedback provided in an embodiment of the present application;

[0059] Figure 5 A flow chart of a glove positioning method based on an infrared sensor provided in an embodiment of the present application;

[0060] Figure 6 A flow chart of a 3D gesture recognition method provided in an embodiment of the present application;

[0061] Figure 7 A flow chart of the glove posture optimization method provided in an embodiment of the present application;

[0062] Figure 8 A schematic diagram of an interface for using gloves to interact with a virtual reality device provided in an embodiment of the present application;

[0063] Fig.9A A schematic diagram of the interaction between the glove and the smooth virtual object provided in the embodiment of the present application;

[0064] Fig. 9B A schematic diagram of pressure feedback when the glove provided in an embodiment of the present application interacts with a smooth virtual object;

[0065] Fig. 10A A schematic diagram of the interaction between the glove provided in the embodiment of the present application and the angular multi-faceted virtual object;

[0066] Fig. 10B A schematic diagram of pressure feedback when the glove provided in an embodiment of the present application interacts with an angular multi-faceted virtual object;

[0067] Fig.11 A structural diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0069] The following is an explanation of the technology used in the interactive method with tactile feedback in the embodiment of the present application.

[0070] 6 Degree of Freedom (6Dof): It is a key technology in VR system. By using infrared sensors and trackers (motion sensors), it can accurately track the position and posture of the user's head and hands, and realize the free movement and interactive operation of objects in the virtual environment. Compared with 3Dof positioning technology, 6Dof positioning technology provides users with a more realistic and freer interactive experience.

[0071] Hand gesture sensing technology: A technology that uses various sensors to achieve more accurate hand gesture sensing, such as the use of inertial measurement units (IMUs, which include gyroscopes, accelerometers), magnetometers, etc. to sense the rotation, tilt, acceleration and other motion information of the hand. By obtaining accurate data on hand gestures, objects and interactive operations in the virtual environment can be controlled more precisely.

[0072] Gesture recognition algorithms: Use machine learning and computer vision algorithms to analyze and identify hand movements. These algorithms accurately identify user gestures and intentions through training models and convert them into control instructions for objects and user interfaces in the virtual environment.

[0073] Haptic feedback technology: Various technologies (such as force feedback devices, vibration feedback devices, tactile sensors, etc.) are used to simulate different tactile sensations to provide real tactile feedback. For example, a force feedback device is used to simulate the real force between the user's hand and the object when the user's hand interacts with the virtual object.

[0074] Wireless communication technology: Use wireless communication technologies such as Bluetooth, Wi-Fi, etc. to communicate with the VR system in real time to ensure fast and stable data transmission between the gloves and the VR system.

[0075] The following is an overview of the design concepts of the embodiments of the present application.

[0076] Since glove interaction frees the hands from restraints, it can provide a freer and more flexible operation method than handle interaction, and provide more realistic tactile feedback than gesture interaction, enhancing the user's sense of immersion. Therefore, it has achieved certain development in the field of VR interaction.

[0077] like Figures 1A to 1F As shown in the figure, there are gloves currently available on the market. Figures 1A to 1D The gloves shown use force feedback devices and tactile sensor technology to provide interactive tactile sensations and force feedback, and the output force is adjustable to simulate the pressure and rigidity of an object; Figure 1E The gloves in the paper use flexible sensors and pneumatic devices to simulate various touch sensations, such as the shape, texture and hardness of objects, but they are complex in structure and expensive. The above five gloves are all made of plastic shells, which have poor fit due to the differences in the hands of different users, and the tactile feedback is only concentrated on the fingertips. Figure 1F The gloves are made of fabric, allowing the hands to maintain a natural shape, and sensors for gesture tracking are deployed on the fingertips. However, they only rely on sensors for single gesture tracking, and the accuracy and latency need to be improved.

[0078] The various gloves with tactile feedback currently on the market still face the following problems:

[0079] Technical limitations: Existing technologies may have technical limitations in sensing, recognizing and processing hand movements, positions and strengths. For example, gesture recognition algorithms may not accurately recognize complex gestures, especially in fast movements and complex scenes. The stability needs to be improved.

[0080] Sensor accuracy and reliability: There may be certain limitations in the accuracy and reliability of sensors. For example, sensors may be affected by noise, interference, and drift, resulting in reduced accuracy in hand posture and position perception.

[0081] Equipment cost and complexity: The high cost and structural complexity of the gloves may lead to insufficient acceptability and scalability of the technology, limiting its popularization in large-scale applications.

[0082] Individual differences among users: Different users have individual differences in hand shape, hand muscle strength, etc. Existing gloves may not be able to adapt to the needs and hand characteristics of different users, thus affecting the personalization and user adaptability of the interactive experience.

[0083] Human-computer interaction design: Existing technologies are insufficient in human-computer interaction design and are unable to provide intuitive, natural and easy-to-use interactive interfaces and operating methods, which affects user acceptance and usage effects.

[0084] In view of this, the embodiment of the present application provides an interactive method with tactile feedback, which uses a camera on a VR device to capture the gestures of the gloves and an infrared sensor to locate and track the human hand. In the positioning and tracking process, the method combines the gesture recognition algorithm with the sensor data to make up for the delay problem of a single gesture recognition algorithm in motion capture, improve the accuracy and efficiency of positioning and tracking, and thus improve the accuracy and response speed of the interaction between the gloves and the virtual reality device. At the same time, the method simulates the tactile feedback when the human hand model in the virtual space contacts the virtual object by adjusting the current of the tactile feedback device on the glove, thereby improving the authenticity of the user's tactile feedback and allowing the user to have a better immersive experience.

[0085] The structure of the gloves used in the embodiments of the present application is as follows Figure 2A As shown in Figure 2C, the glove is equipped with 17 infrared sensors (such as infrared LED lights) for tracking the position of the hand at the fingertips and joints of each finger and at the metacarpal bones of the back of the hand. Figure 2A As shown in the circle in the back view of the hand, the gesture recognition algorithm deployed on the VR device can recognize the infrared image of the glove when the infrared sensor is turned on, monitor the hand posture in real time, accurately track the hand position, determine the hand movement, and accurately transmit the user's hand operation and movement to the virtual environment. At the same time, the glove is equipped with 19 tactile feedback devices with vibration and pressure feedback (such as linear resonant actuators (LAR), referred to as LRA motors) on the fingertips and joints of each finger on the palm side and on the palm metacarpal bones. Figure 2BAs shown in the squares in the palm view, these LRA motors can provide tactile feedback that matches the touch, collision or pressure events in the virtual environment. By controlling the vibration frequency and intensity of the LRA motors, users can feel the texture, shape and strength of different objects. Figure 2C shown.

[0086] like Figure 2A As shown, the glove is wirelessly connected to the VR device via a battery installed on the back of the wrist. Compared with a wired connection, this method makes the movement of the glove more flexible and the user can move freely without being restricted by wires.

[0087] Not shown, a plurality of motion sensors, such as a gyroscope, an accelerometer, a magnetometer, etc., are also installed on the back of the wrist of the glove to sense motion information such as rotation, tilt and acceleration of the hand.

[0088] In addition, the gloves are designed with two layers, the outer layer uses a lightweight elastic fabric (such as spandex), which makes the gloves light, soft and breathable, and can fit the hands well, providing a comfortable wearing feeling. The inner layer uses a detachable and washable sweat-absorbent material (such as nylon, polyester, etc.), so that users can regularly clean and replace the inner layer, ensuring hygiene while improving wearing comfort.

[0089] It should be noted that FIG. 2A to FIG. 2C This is just an example, and the embodiment of the present application does not impose any restrictive requirements on the number and position of the infrared sensors and tactile feedback devices, as well as the material of the gloves. For example, in addition to using lightweight stretch fabrics for the outer layer of the gloves, other soft and comfortable materials (such as flexible silicone, etc.) can also be considered to meet the comfort and flexibility requirements of different users.

[0090] In one example, the battery on the glove adopts an efficient energy management solution, such as low-power battery technology, rechargeable function and energy-saving mode, so as to ensure that the glove has a reliable power supply during use, improve the convenience and durability of the glove, and extend the use time of the glove.

[0091] In one example, components on the gloves (such as infrared sensors, tactile feedback devices, motion sensors, batteries, etc.) are detachable, adjustable, and replaceable, making the gloves expandable and upgradeable to meet future technological developments and changes in user needs, facilitating subsequent upgrades and maintenance, and allowing users to adjust and replace them according to their hand shape and personal preferences to obtain a more comfortable and better-fitting interactive experience.

[0092] For example, in scenarios where high interaction accuracy is required, higher-precision sensors can be replaced to improve the accuracy and precision of hand posture, position, and force perception, and enhance the sensitivity and authenticity of the interaction.

[0093] For another example, the LRA motor can be replaced with a multimodal tactile feedback device that combines multiple tactile feedback technologies (such as force feedback, vibration, temperature, etc.) to simulate a richer tactile experience and further enhance the user's sense of immersion.

[0094] In one example, the gloves are compatible with different types of devices. In addition to being able to connect to VR devices, they can also be connected to display devices such as PCs, smart phones, and game consoles, thereby using the display devices as interactive interfaces to achieve seamless connection between the gloves and the virtual environment.

[0095] In one example, users can easily perform rich personalized configuration and updates on the components of the gloves through a software interface or cloud service, allowing users to customize the functions and parameters of the gloves according to their preferences and needs, achieving highly customized functions of the gloves.

[0096] In the embodiments of the present application, through the improvement of the gloves, the problems existing in the existing gloves in terms of interaction mode, tactile feedback, accuracy, adaptability and user experience are solved, and the performance, comfort and usability of the gloves are effectively improved, making the gloves a more attractive and realistic virtual reality interaction tool, which can provide an immersive tactile experience and create a more realistic and interactive virtual interaction environment for users.

[0097] Based on the above gloves, the interactive architecture with tactile feedback provided by the embodiment of the present application is shown in the figure Figure 3As shown, the glove is equipped with multiple infrared sensors, multiple motion feedback devices and multiple motion sensors, and the virtual reality device is equipped with a multi-camera (usually a binocular or quadruple fisheye camera). The virtual reality device and the glove interact through a wireless connection. During the interaction, the virtual reality device collects the infrared image of the glove when the multiple infrared sensors on the glove are on through the camera, and extracts the infrared spots in the infrared image of the glove. According to the position structure of the multiple infrared sensors on the glove, the position deviation between the extracted infrared spots can be obtained, so as to correct the position of each infrared spot and obtain the spot reflecting the skeleton joint. Based on the corrected infrared spots, the initial posture and orientation of the glove are calculated by using a computer vision algorithm, so as to complete the positioning process based on the infrared sensor. At the same time, the positioning result of the infrared sensor can be used as a priori for 3D gesture recognition based on deep learning, which can effectively improve the accuracy and efficiency of 3D gesture recognition compared with single gesture recognition. Among them, in the process of 3D gesture recognition, the initial position and orientation determined by the infrared sensor can be used as motion information for 3D gesture recognition to provide the area of ​​the glove in the image, and the skeleton model is obtained by feature extraction and key point detection of the area, so as to accurately and quickly recognize the 3D gesture. In order to improve the positioning accuracy of the glove, the measurement results of the motion sensor and the positioning results of the infrared sensor can be multimodal data fused to optimize the initial position of the glove and obtain a high-precision target position. Combined with the recognized 3D gesture and the orientation of the glove, the glove model that interacts with the virtual object is displayed in the virtual screen, and the parameters of the tactile feedback device are adjusted according to the properties of the virtual object, so that the tactile feedback device vibrates according to the parameters. In this way, the user can experience the tactile sense of the interaction between the hand and the object, improve the authenticity of the interaction, and obtain a more immersive interactive experience.

[0098] Figure 4 The process of an interactive method with tactile feedback provided in an embodiment of the present application is exemplarily shown. The process is executed by a virtual reality device and mainly includes the following steps:

[0099] S401: Acquire an infrared image of the glove taken by a camera when the infrared sensor is on, and acquire motion measurement data of the glove collected by a motion sensor.

[0100] In one example, the camera on the virtual reality device uses short exposure to capture an infrared image of the glove when the infrared sensor on the glove lights up, thereby eliminating interference from ambient light and background, improving the accuracy of subsequent infrared spot detection, and further improving the accuracy of glove positioning.

[0101] Since the gloves are equipped with motion sensors such as gyroscopes, accelerometers, and magnetometers, during the interaction between the gloves and virtual reality devices, motion measurement data such as angular velocity and acceleration of the hand movements can be collected in real time.

[0102] S402: Detect the infrared spot in the infrared image of the glove, and determine the initial position and orientation of the glove in the global coordinate system in combination with the intrinsic parameters of the camera obtained through calibration.

[0103] In one example, a virtual reality device uses an image processing algorithm to obtain the position of the infrared spot of the infrared sensor on the glove in the image from the infrared image of the glove captured by the camera. Combined with the memory book of the camera obtained through calibration, the initial position of the glove in the global coordinate system (i.e., the world coordinate system) (including the translation matrix t representing the position of the glove and the rotation matrix r representing the posture of the glove) and the orientation relative to the infrared spot can be measured, thereby completing the glove positioning process based on the infrared sensor.

[0104] In the specific implementation, the glove positioning process based on infrared sensors can be found in Figure 5 , mainly includes the following steps:

[0105] S4021: Use a feature point detection algorithm to detect the infrared image of the glove to obtain the initial 2D coordinates of each infrared spot in the infrared image of the glove.

[0106] An optional implementation is to use a feature point detection algorithm in image processing (such as Harris corner detection or FAST corner detection, etc.) to detect each infrared spot in the infrared image of the glove, and perform preprocessing operations such as calibration, filtering and data alignment on each infrared spot to ensure the accuracy and consistency of the data, thereby obtaining the initial 2D coordinates of each infrared spot in the image coordinate system, recorded as (u, v).

[0107] S4022: Optimize each initial 2D coordinate according to the position structure of the infrared sensor on the glove to obtain the target 2D coordinate of each infrared spot.

[0108] When the gloves leave the factory, the installation position of each infrared sensor is known. Therefore, the position deviation between each infrared sensor can be obtained according to the position structure of each infrared sensor. In this way, the initial 2D coordinates of each infrared spot detected can be optimized by using the position deviation to obtain more accurate target 2D coordinates of each infrared spot in the infrared image of the glove, thereby improving the positioning accuracy of the glove.

[0109] S4023: Obtain the 3D coordinates of each infrared sensor on the glove, and determine the initial posture of the glove in the global coordinate system by combining the intrinsic parameters of the camera and the target 2D coordinates of each infrared spot.

[0110] In one example, the intrinsic parameters of the camera can be obtained by Zhang Zhengyou calibration method, and the 3D coordinates of each infrared sensor on the glove can be obtained as a known quantity, denoted as (X w , Y w , Zw ), combined with the 2D coordinates of each infrared sensor in the infrared image of the handle, the PnP algorithm can be used to determine the initial pose [r, t] of the glove in the global coordinate system.

[0111] In the specific implementation, according to the intrinsic parameters of the camera, the target 2D coordinates of each infrared spot in the infrared image of the glove are converted to the camera coordinate system to obtain the three-dimensional camera coordinates, which are recorded as (X c , Y c , Z c ), and then calculate the rotation matrix t and translation matrix t according to the camera coordinates and the 3D coordinates of each infrared sensor to obtain the initial position of the glove in the global coordinate system. The formula is as follows:

[0112]

[0113]

[0114] Where f represents the focal length of the camera, (c x , c y ) represents the optical center of the camera, Z c is the transformation coefficient.

[0115] S4024: Determine the camera coordinates of each infrared spot in the camera coordinate system according to the target 2D coordinates of each infrared spot and the intrinsic parameters of the camera, normalize each camera coordinate, and determine the orientation of the glove.

[0116] In specific implementation, the target 2D coordinates of each infrared spot are converted to the camera coordinate system using the intrinsic parameters of the camera, and then the camera coordinates are normalized into unit vectors. Each unit vector represents the direction of the corresponding infrared spot, and then the direction of the glove is obtained using the direction of each infrared spot. The formula of the normalization process is expressed as:

[0117]

[0118] u x =X c / L,u y =Y c / L,u z =Z c / L Formula 4

[0119] Among them, (u x ,u y ,u z ) is a unit vector representing the direction.

[0120] In one example, when positioning the glove based on the infrared sensor, the shape and size of the infrared spot may also be considered to further improve the positioning accuracy of the glove.

[0121] After obtaining the positioning result of the infrared sensor on the glove, the positioning result can be used as a priori for 3D gesture recognition, thereby improving the speed and accuracy of 3D gesture recognition.

[0122] S403: Recognize the 3D gesture of the glove according to the initial posture and the infrared spots in the infrared image of the glove.

[0123] In one example, a machine learning algorithm can be used to train a gesture recognition classifier to map the hand movement and posture information contained in the infrared image of the glove captured by the camera into a 3D gesture with specific instructions or commands, thereby realizing a visual positioning process based on an infrared sensor.

[0124] Among them, when training the gesture recognition classification model, a large number of training samples with gesture labels are prepared in advance, and each gesture label represents a gesture type, such as 1 represents the gesture type of open palm, 2 represents the gesture type of clenched fist, 3 represents the gesture type of index finger up, 4 represents the gesture type of index finger down, etc., so that the gesture recognition classifier can accurately obtain the corresponding gesture label according to the features extracted from the glove infrared image.

[0125] For the specific 3D gesture recognition process, see Figure 6 , mainly includes the following steps:

[0126] S4031: Obtain a target area of ​​the hand in the infrared image of the glove according to the initial posture.

[0127] In one example, the infrared image of the glove captured by the camera when the infrared sensor on the glove is on is preprocessed by signal processing and filtering to eliminate noise and interference in the infrared image of the glove and improve the accuracy of 3D gesture recognition. Then, the initial position of the glove in the global coordinate system determined by each infrared spot can be used to determine the approximate target area of ​​the glove in the infrared image of the glove, thereby reducing the computational complexity of feature extraction during 3D gesture recognition and improving the speed of 3D gesture recognition.

[0128] S4032: Extract features of the target area to obtain hand state features, and detect hand key points of the target area to obtain hand skeleton features.

[0129] In one example, hand state features for gesture recognition are extracted from a target area, including but not limited to finger bending angles, palm orientation, relative positions between fingers, hand contours, hand shapes, etc., and hand key point detection is performed on the target area using a MediaPipe or dlib algorithm to obtain hand skeleton features.

[0130] S4033: Using each infrared light spot as a reference point, obtaining hand change characteristics of the glove relative to each reference point.

[0131] In one example, each infrared spot detected in the infrared image of the glove can be used as a reference point for gesture recognition to obtain the hand change characteristics of the glove (ie, gesture) relative to each reference point, thereby making it easier to detect the movement and posture of the 3D gesture.

[0132] S4034: Determine the gesture label of the glove according to the hand state features, the hand skeleton features and the hand change features, and obtain the 3D gesture of the glove.

[0133] The hand state features, hand skeleton features and hand change features are spliced ​​to generate a feature vector, denoted as F = f(G), G = {g_1, g_2, ..., g_N}, g_i represents the gesture data represented by the i-th infrared sensor, and the feature vector is input into the trained gesture recognition classifier to determine the current gesture label of the glove, thereby obtaining the current 3D gesture.

[0134] It should be noted that the embodiments of the present application do not restrict the deep learning algorithm used by the gesture recognition classifier, including but not limited to support vector machine (SVM), decision tree, random forest or deep learning (such as convolutional neural network (CNN) etc.) methods. By adopting advanced gesture recognition algorithms, the accuracy and response speed of gesture recognition can be improved, and more accurate interactive operations and more natural user experience can be achieved.

[0135] During the 3D gesture recognition process of the embodiment of the present application, the infrared spot provides auxiliary information for gesture recognition. The initial position of the glove determined based on the infrared spot can provide the position of the glove in real time for 3D gesture recognition. This means that at each time point, the virtual reality device can obtain the approximate area of ​​the current glove in the infrared image of the glove, without the need for continuous image analysis, thereby reducing the delay of gesture recognition. In addition, by obtaining the approximate area of ​​the glove in the infrared image of the glove, feature extraction is performed in the target area. Compared with feature extraction in the entire image, the feature search space of gesture recognition is reduced, the complexity and amount of feature extraction are reduced, and the delay of gesture recognition is further reduced. On the other hand, by using the infrared spot as the reference point for gesture recognition, the change of the gesture relative to the reference point can be obtained, thereby helping the gesture recognition classifier to detect 3D gestures more easily and improve the accuracy of gesture detection.

[0136] S404: Optimize the initial posture according to the motion measurement data to determine the target posture of the glove in the global coordinate system.

[0137] Motion sensors such as gyroscopes, accelerometers and magnetometers are also installed on the wrist of the back of the glove, which can obtain motion measurement data of hand movement in real time. The accuracy and consistency of the data are ensured by preprocessing operations such as calibration, filtering and data alignment of the motion measurement data, and a data fusion algorithm is used to fuse the motion measurement data collected by each motion sensor, thereby improving the positioning result of the motion sensor.

[0138] In one example, the motion measurement data includes acceleration and angular velocity that represent hand motion information, and the posture of the glove can be estimated, so that the initial posture based on infrared sensor positioning can be optimized according to the estimation result to obtain a more accurate positioning result.

[0139] It should be noted that the embodiments of the present application do not impose restrictive requirements on the optimization algorithm, for example, Kalman filtering, unscented Kalman filtering, etc. can be used. Kalman filtering is a data fusion method suitable for linear systems, which can fuse the positioning results of motion sensors and infrared sensors, and can dynamically correct the glove posture estimated based on motion measurement data. At the same time, through posture estimation and data fusion, the influence of the uncertainty of infrared spots collected under different states and the uncertainty of motion measurement data on the positioning results can be solved, effectively improving the positioning accuracy of the gloves, thereby improving the accuracy of the interaction between the gloves and the virtual reality device.

[0140] Taking Kalman filtering as an example, the specific data fusion process can be found in Figure 7 , mainly includes the following steps:

[0141] S4041: Initialize the state vector, state covariance matrix, noise covariance matrix, measurement noise covariance matrix, state transfer matrix and measurement matrix of the Kalman filter.

[0142] In one example, the motion measurement data usually includes acceleration and angular velocity. By integrating the angular velocity, the posture of the glove can be estimated in real time, which is expressed as (θ x ,θ y ,θ z ), by integrating the acceleration, the position of the glove can be estimated in real time, denoted as (X t , Y t , Z t ). The integration result of acceleration and angular velocity is defined as the state vector of Kalman filtering, denoted as X k =[X t , Y t , Z t ,θ x ,θ y ,θ z] represents the position and posture (i.e., position and attitude) of the glove in the global coordinate system determined based on the motion sensor.

[0143] At the same time, define the Kalman filter state covariance matrix P k , noise covariance matrix Q, measurement noise covariance matrix R, state transfer matrix A and measurement matrix H, and initialize them.

[0144] In some embodiments, when the positioning of the glove based on the infrared sensor fails, such as when the palm of the glove is facing the camera, no infrared spot is collected or the number of infrared spots is insufficient, at this time, the real-time posture estimation result based on the motion sensor can be used as auxiliary information for 3D gesture recognition.

[0145] S4042: Predict the state vector at the current moment based on the state transfer matrix and the state vector at the previous moment.

[0146] The state transfer matrix is ​​used to predict the motion state at the next moment. The formula is expressed as:

[0147] X′ k =A*X k Formula 5

[0148] Among them, X k represents the state vector at the previous moment, X′ k Represents the state vector at the current moment.

[0149] S4043: Predict the state covariance matrix at the current moment according to the noise covariance matrix, the state transfer matrix and the state covariance matrix at the previous moment.

[0150] The noise covariance matrix is ​​used to predict the state covariance matrix at the next moment. The formula is expressed as:

[0151] P′ k =A*P k *A T +Q Formula 6

[0152] Among them, P k Represents the state covariance matrix of the previous moment, P′ k Represents the state covariance matrix at the current moment.

[0153] S4044: Calculate the Kalman filter gain according to the state covariance matrix, measurement matrix and measurement noise covariance matrix at the current moment.

[0154] Specifically, the calculation formula of the Kalman filter gain K is:

[0155] K=P′ k *H T *(H*P′k *H T +R) -1 Formula 7

[0156] S4045: Optimize the initial posture according to the current state vector, Kalman filter gain and measurement matrix to obtain the target posture of the glove in the global coordinate system.

[0157] Get the initial pose of the glove determined by the infrared sensor, denoted as The initial position and state covariance matrix of the infrared sensor positioning are corrected using the current glove position and posture estimated based on motion measurement data. The formula is expressed as:

[0158] S′ k =X′ k +K(S k -H*X′ k ) Formula 8

[0159] P k =(IK*H)*P′ k Formula 9

[0160] Among them, S′ k is the target pose of the fused glove in the global coordinate system.

[0161] In the process of optimizing the posture of the glove in the global coordinate system, the positioning results of the motion sensor and the infrared sensor are fused to check the redundancy between different data sources, provide more comprehensive glove motion status information, and effectively improve the reliability and accuracy of glove positioning for accurate interaction in a virtual environment.

[0162] S405: Displaying a glove model that interacts with a virtual object in a virtual screen according to the 3D gesture, target position and orientation.

[0163] In one example, the position of the glove model in the virtual screen is determined according to the target position of the glove in the global coordinate system, and the glove model is displayed at the target position according to the recognized 3D gesture according to the target posture and orientation of the glove, so that the user can interact with the objects in the virtual screen by controlling the glove, such as Figure 8 shown.

[0164] S406: adjusting parameters of the tactile feedback device according to the properties of the virtual object, so that the glove provides the target user with a tactile sense of contact between the hand and the virtual object according to the parameters.

[0165] In actual scenes, when the user's hand touches a real object, he can feel the temperature of the object, the force of the object on the hand, etc. In order to provide an interactive experience consistent with the real environment, multiple tactile feedback devices are installed on the gloves. The virtual reality device adjusts the parameters of each tactile feedback device according to the properties of the virtual object touched by the glove model in the virtual screen, including but not limited to the shape, material, size, etc. of the object, so that each tactile sensor provides the tactile sensation of the glove touching the virtual object, thereby improving the authenticity of the virtual interaction and giving the user a better sense of immersion.

[0166] In one example, taking the tactile feedback device as an LRA motor, the LRA motor is a vibration motor that works on the principle of resonance. It consists of an eccentric shaft with a fixed mass and an elastic component (such as a spring). When current passes through the coil, the generated magnetic field interacts with the permanent magnet, causing the eccentric shaft to vibrate near the resonance frequency. Therefore, the virtual reality device achieves adjustable vibration feedback by adjusting the current value input to the tactile feedback device, causing the LRA motor to produce different vibrations, thereby providing the glove with the force when it contacts different virtual objects.

[0167] Optionally, when the glove model contacts a virtual object, the virtual reality device can change the vibration amplitude of the eccentric shaft by adjusting the current amplitude input to the LRA motor. The larger the current amplitude, the stronger the vibration of the LRA motor, thereby achieving vibration feedback of different intensities and simulating the feeling of an object applying different amounts of pressure to the hand.

[0168] Optionally, when the glove model contacts a virtual object, the virtual reality device can change the vibration frequency of the LRA motor by adjusting the frequency of the current input to the LRA motor. The greater the current frequency, the faster the LRA motor vibrates, thereby achieving vibration feedback of different frequencies and simulating the feeling of an object applying pressure of different speeds to the hand.

[0169] In one example, the force adjustment range of the LRA motor can be set by adjusting the current amplitude of the LRA motor and the mechanical characteristics of the LRA motor (such as torque, speed, etc.). For example, multiple LRA motors on each finger are set to provide 8 pounds of force feedback.

[0170] It should be noted that the specific tactile adjustment details and parameters depend on the design and manufacturing process of the LRA motor. During the design phase, the parameters are determined through detailed engineering analysis, testing and optimization to ensure that the LRA motor can provide the required force and performance. In addition, the number and position of the LRA motor can be appropriately installed and adjusted according to the structure of the glove and the structure of the user's hand to achieve an accurate and comfortable tactile feedback experience.

[0171] In some embodiments, the tactile characteristics and force feedback data of different objects can be obtained, and a tactile database can be established. Combined with simulation technology, a more accurate and realistic tactile feeling can be achieved, allowing users to interact more realistically with virtual objects in a virtual environment.

[0172] In the embodiments of the present application, by adjusting the parameters of the LRA motor, the user can experience different degrees of friction and squeezing force on the hand when contacting different virtual objects.

[0173] like Fig.9A As shown in Figure 2, when the glove model contacts a virtual object with a smooth surface, the pressure feedback output by the LRA motor on the glove is as follows: Fig. 9B As shown, when stroking the virtual sphere, the pressure felt by the palm tends to be gentle. At this time, the force applied to different areas of the palm is relatively uniform, which is consistent with the tactile sensation of stroking a real sphere.

[0174] like Fig. 10A As shown in Figure 2, when the glove model contacts an angular virtual object, the pressure feedback output by the LRA motor on the glove is as follows: Fig. 10B As shown, when stroking the virtual polyhedron, the pressure felt by the palm is uneven. At this time, the forces on different areas of the palm are uneven, which has the same tactile feeling as stroking a real polyhedron.

[0175] In some embodiments, an intelligent control system may also be integrated into the glove, which may adaptively adjust the tactile feedback and hand motion regulation provided by the tactile feedback device through an optimization algorithm or a machine learning algorithm, thereby providing a smarter and more personalized user experience.

[0176] In some embodiments, when using gloves to interact with a virtual reality device, the virtual reality device can provide an intuitive and simple user interface so that the user can easily configure and control the functions of the gloves according to their own needs. For example, the user can customize the tactile feedback mode, feedback force intensity, etc. in the user interface through buttons or voice control, or switch between different interaction modes (such as switching from glove interaction to handle interaction).

[0177] The interactive method with tactile feedback provided in the embodiment of the present application achieves precise positioning of the glove and 3D gesture recognition by installing multiple infrared sensors and multiple motion sensors on the back of the glove, thereby improving the accuracy and response speed of the interaction between the glove and the virtual reality device. In addition, by installing multiple tactile feedback devices on the palm side of the glove, the limitations of the handle interaction and gesture interaction methods are solved at the same time, providing a more flexible operation method and more realistic tactile feedback, and effectively enhancing the user's immersive experience.

[0178] Based on the same technical concept, an embodiment of the present application provides a virtual reality device. Based on the virtual reality device, the above-mentioned interaction method with tactile feedback can be implemented and the same technical effect can be achieved, which will not be repeated here.

[0179] See also Fig.11 , the virtual reality device includes a processor 1101, a memory 1102, a display 1103, a camera 1104 and a communication interface 1105, and the communication interface 1105, the camera 1104, the display 1103, the memory 1102 and the processor 1101 are connected via a bus 1106;

[0180] The communication interface 1105 is used to interact with the gloves, which are equipped with multiple infrared sensors, multiple tactile feedback devices and multiple motion sensors;

[0181] The memory 1102 stores a computer program, and the processor 1101 performs the following operations according to the computer program:

[0182] Acquire an infrared image of the glove taken by the camera 1104 when the infrared sensor is on, and acquire motion measurement data of the glove collected by the motion sensor;

[0183] Detect the infrared spot in the infrared image of the glove, and determine the initial position and orientation of the glove in the global coordinate system by combining the intrinsic parameters of the camera obtained through calibration;

[0184] According to the initial posture and the infrared spots in the infrared image of the glove, the 3D gesture of the glove is recognized;

[0185] The initial posture is optimized according to the motion measurement data to determine the target posture of the glove in the global coordinate system;

[0186] Displaying a glove model interacting with a virtual object in a virtual picture displayed on the display 1103 according to the 3D gesture, target posture and orientation;

[0187] According to the properties of the virtual object, the parameters of the tactile feedback device are adjusted so that the glove provides the target user with a tactile sense of contact between the hand and the virtual object according to the parameters.

[0188] Optionally, the processor 1101 detects the infrared spot in the infrared image of the glove and combines the internal and external parameters of the camera obtained by calibration to determine the initial position and orientation of the glove in the global coordinate system. The specific operations are:

[0189] The infrared image of the glove is detected using a feature point detection algorithm to obtain the initial 2D coordinates of each infrared spot in the infrared image of the glove;

[0190] According to the position structure of the infrared sensor on the glove, each initial 2D coordinate is optimized to obtain the target 2D coordinate of each infrared spot;

[0191] Obtain the 3D coordinates of each infrared sensor on the glove, and determine the initial pose of the glove in the global coordinate system by combining the intrinsic parameters of the camera and the target 2D coordinates of each infrared spot;

[0192] According to the target 2D coordinates of each infrared spot and the intrinsic parameters of the camera, the camera coordinates of each infrared spot in the camera coordinate system are determined, and each camera coordinate is normalized to determine the orientation of the glove.

[0193] Optionally, the processor 1101 recognizes the 3D gesture of the glove according to the initial posture and each infrared spot in the infrared image of the glove, and the specific operation is:

[0194] According to the initial posture, the target area of ​​the hand in the infrared image of the glove is obtained;

[0195] Extract features of the target area to obtain hand state features, and detect hand key points in the target area to obtain hand skeleton features;

[0196] Each infrared spot is used as a reference point to obtain the hand change characteristics of the glove relative to each reference point;

[0197] According to the hand state features, hand skeleton features and hand change features, the gesture label of the glove is determined to obtain the 3D gesture of the glove.

[0198] Optionally, the motion measurement data includes acceleration and angular velocity. The processor 1101 optimizes the initial posture according to the motion measurement data to determine the target posture of the glove in the global coordinate system. The specific operations are:

[0199] Initialize the state vector, state covariance matrix, noise covariance matrix, measurement noise covariance matrix, state transfer matrix and measurement matrix of the Kalman filter; wherein the state vector is determined according to the integral of acceleration and angular velocity;

[0200] According to the state transfer matrix and the state vector at the previous moment, predict the state vector at the current moment;

[0201] According to the noise covariance matrix, the state transfer matrix and the state covariance matrix of the previous moment, the state covariance matrix of the current moment is predicted;

[0202] Calculate the Kalman filter gain according to the state covariance matrix, measurement matrix and measurement noise covariance matrix at the current moment;

[0203] According to the current state vector, Kalman filter gain and measurement matrix, the initial pose is optimized to obtain the target pose of the glove in the global coordinate system.

[0204] Optionally, the glove is divided into two layers, an inner layer and an outer layer, and a plurality of infrared sensors are installed on the fingertips and joints of each finger on the back of the hand and on the metacarpal bones of the back of the hand on the outer layer of the glove;

[0205] A plurality of tactile feedback devices are installed on the fingertips and joints of each finger and the metacarpal bones of the palm on the palm side of the outer layer of the glove; a plurality of motion sensors are installed on the wrist on the back side of the inner layer of the glove.

[0206] Optionally, the outer layer of the gloves is made of lightweight stretch fabric, and the inner layer of the gloves is made of sweat-absorbent material, and the inner layer of the gloves can be removed, washed and replaced.

[0207] Optionally, the tactile feedback device is a linear resonance motor, and the parameters of the tactile feedback device include at least one of input current amplitude, input current frequency and mechanical characteristics; wherein the input current amplitude and mechanical characteristics are used to change the vibration intensity of the linear resonance motor; and the input current frequency is used to change the vibration frequency of the linear resonance motor.

[0208] In the embodiment of the present application, a plurality of infrared sensors, a plurality of tactile feedback devices and a plurality of motion sensors are installed on the gloves that interact with the virtual reality device. In this way, the virtual reality device can obtain the initial position and orientation of the gloves in the global coordinate system through the visual positioning technology according to the infrared spots detected in the infrared image of the gloves taken by the camera and the internal parameters of the camera, and the initial position and infrared spots of the gloves can be used as the priori for gesture recognition of the infrared image of the gloves, thereby improving the efficiency and accuracy of gesture recognition, and further improving the response speed and accuracy of the interaction between the gloves and the virtual reality device; at the same time, the virtual reality device estimates the position of the gloves using the motion measurement data collected by the motion sensor on the gloves, and optimizes the initial position of the infrared sensor positioning according to the estimation result, effectively improving the positioning accuracy of the gloves. Furthermore, according to the optimized target position, the recognized 3D gesture and the orientation of the gloves, the glove model that interacts with the virtual object can be displayed in the virtual screen, and the tactile feedback device on the gloves can give the target user's hand the tactile sense of interacting with the virtual object, providing the target user with an immersive experience like being in the scene.

[0209] in, Fig.11The memory 1102 in the embodiment may be a volatile memory, such as a random-access memory (RAM); the memory 1102 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1102 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor 1101 may include one or more central processing units (CPUs) or a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0210] It should be noted that Fig.11 This is just an example, and provides the necessary hardware for a virtual reality device to execute the steps of an interactive method with tactile feedback provided in an embodiment of the present application. If not shown, the virtual reality device may also include hardware of a conventional VR device such as a speaker, a pickup, a power supply, and an IMU.

[0211] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete an interactive method with tactile feedback in the aforementioned embodiment.

[0212] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute an interaction method with tactile feedback in the aforementioned embodiment.

[0213] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0214] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram, and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes in the block diagram.

[0215] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0217] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An interactive method with tactile feedback, characterized in that: Applied to a virtual reality device, the virtual reality device interacts with a headgear, the gloves are equipped with multiple infrared sensors, multiple tactile feedback devices and multiple motion sensors, and the method includes: Acquire an infrared image of the glove taken by a camera when the infrared sensor is on, and acquire motion measurement data of the glove collected by the motion sensor; Detecting the infrared spot in the infrared image of the glove, and determining the initial position and orientation of the glove in the global coordinate system in combination with the intrinsic parameters of the camera obtained through calibration; Recognize the 3D gesture of the glove according to the initial posture and each infrared spot in the infrared image of the glove; Optimizing the initial posture according to the motion measurement data to determine the target posture of the glove in a global coordinate system; Displaying a glove model interacting with a virtual object in a virtual screen according to the 3D gesture, the target posture, and the orientation; According to the attributes of the virtual object, the parameters of the tactile feedback device are adjusted so that the glove provides the target user with a tactile sense of contact between the hand and the virtual object according to the parameters.

2. The method according to claim 1, characterized in that The detecting of the infrared spot in the infrared image of the glove and determining the initial position and orientation of the glove in the global coordinate system by combining the internal and external parameters of the camera obtained by calibration include: Using a feature point detection algorithm to detect the infrared image of the glove, and obtaining the initial 2D coordinates of each infrared spot in the infrared image of the glove; According to the position structure of the infrared sensor on the glove, each initial 2D coordinate is optimized to obtain the target 2D coordinate of each infrared spot; Obtaining the 3D coordinates of each infrared sensor on the glove, and determining the initial position and posture of the glove in the global coordinate system by combining the intrinsic parameters of the camera and the target 2D coordinates of each infrared spot; According to the target 2D coordinates of each infrared light spot and the intrinsic parameters of the camera, the camera coordinates of each infrared light spot in the camera coordinate system are determined, and each camera coordinate is normalized to determine the orientation of the glove.

3. The method according to claim 1, characterized in that The step of recognizing the 3D gesture of the glove according to the initial posture and each infrared spot in the infrared image of the glove includes: According to the initial posture, obtaining a target area of ​​the hand in the infrared image of the glove; Performing feature extraction on the target area to obtain hand state features, and performing hand key point detection on the target area to obtain hand skeleton features; Taking each infrared light spot as a reference point, obtaining the hand change characteristics of the glove relative to each reference point; According to the hand state feature, the hand skeleton feature and the hand change feature, a gesture label of the glove is determined to obtain a 3D gesture of the glove.

4. The method according to claim 1, characterized in that The motion measurement data includes acceleration and angular velocity, and the initial posture is optimized according to the motion measurement data to determine the target posture of the glove in the global coordinate system, including: Initialize a state vector, a state covariance matrix, a noise covariance matrix, a measurement noise covariance matrix, a state transfer matrix, and a measurement matrix of a Kalman filter; wherein the state vector is determined based on the integral of the acceleration and the angular velocity; Predicting the state vector at the current moment based on the state transfer matrix and the state vector at the previous moment; Predicting the state covariance matrix at the current moment according to the noise covariance matrix, the state transfer matrix and the state covariance matrix at the previous moment; Calculating a Kalman filter gain according to the state covariance matrix, the measurement matrix and the measurement noise covariance matrix at the current moment; The initial posture is optimized according to the state vector at the current moment, the Kalman filter gain and the measurement matrix to obtain a target posture of the glove in a global coordinate system.

5. The method according to any one of claims 1 to 4, characterized in that The gloves are divided into two layers, inner and outer layers, and the plurality of infrared sensors are installed on the fingertips and joints of each finger on the back of the hand of the outer layer of the gloves, as well as on the back of the hand metacarpal bones; The multiple tactile feedback devices are installed on the fingertips and joints of each finger on the palm side of the outer layer of the glove, as well as on the palm metacarpal bones; The plurality of motion sensors are installed on the wrist of the back of the hand on the inner layer of the glove.

6. The method according to claim 5, characterized in that The outer layer of the gloves is made of lightweight stretch fabric, the inner layer of the gloves is made of sweat-absorbent material, and the inner layer of the gloves can be disassembled, washed and replaced.

7. The method according to claim 5, characterized in that The tactile feedback device is a linear resonance motor, and the parameters of the tactile feedback device include at least one of an input current amplitude, an input current frequency, and a mechanical characteristic; wherein the input current amplitude and the mechanical characteristic are used to change the vibration intensity of the linear resonance motor; and the input current frequency is used to change the vibration frequency of the linear resonance motor.

8. A virtual reality device, characterized in that: It includes a processor, a memory, a display, a camera and a communication interface, wherein the communication interface, the camera, the display, the memory and the processor are connected via a bus; The communication interface is used to interact with the glove, and the glove is equipped with multiple infrared sensors, multiple tactile feedback devices and multiple motion sensors; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquire an infrared image of the glove taken by the camera when the infrared sensor is on, and acquire motion measurement data of the glove collected by the motion sensor; Detecting the infrared spot in the infrared image of the glove, and determining the initial position and orientation of the glove in the global coordinate system in combination with the intrinsic parameters of the camera obtained through calibration; Recognize the 3D gesture of the glove according to the initial posture and each infrared spot in the infrared image of the glove; Optimizing the initial posture according to the motion measurement data to determine the target posture of the glove in a global coordinate system; Displaying a glove model interacting with a virtual object in a virtual picture displayed on the display according to the 3D gesture, the target posture and the orientation; According to the attributes of the virtual object, the parameters of the tactile feedback device are adjusted so that the glove provides the target user with a tactile sense of contact between the hand and the virtual object according to the parameters.

9. The virtual reality device according to claim 8, characterized in that: The gloves are divided into two layers, inner and outer layers, and the plurality of infrared sensors are installed on the fingertips and joints of each finger on the back of the hand of the outer layer of the gloves and on the back of the hand metacarpal bones; The multiple tactile feedback devices are installed on the fingertips and joints of each finger on the palm side of the outer layer of the glove and on the palm metacarpal bones; The plurality of motion sensors are installed on the wrist of the back of the hand on the inner layer of the glove.

10. A glove for interacting with a virtual reality device, characterized in that: The glove is divided into two layers, an inner layer and an outer layer. A plurality of infrared sensors are installed on the fingertips and joints of each finger and on the metacarpal bones of the back of the hand on the outer layer of the glove. A plurality of tactile feedback devices are installed on the fingertips and joints of each finger and on the metacarpal bones of the palm side of the outer layer of the glove. A plurality of motion sensors are installed on the wrist of the back of the hand on the inner layer of the glove. The outer layer of the glove is made of lightweight stretch fabric, and the inner layer of the glove is made of sweat-absorbent material. The inner layer of the glove can be disassembled, washed and replaced.

Images