Interaction Method, Electronic Device, and Interaction System

Abstract

The present application provides an interaction method, an electronic device and an interaction system, relating to the field of virtual reality technology. Among them, the method includes: a head-mounted device displays a virtual keyboard; a hand-worn device responds to a user's input trigger operation, and sends hand detection data to the head-mounted device according to the detected target sensor signal; after the head-mounted device determines that an input trigger event has occurred according to the hand detection data, it determines an initial click position of a target finger relative to the virtual keyboard according to a hand image collected by a camera; the hand-worn device sends hand motion tracking data collected by a motion sensor to the head-mounted device; the head-mounted device responds to the user's input end operation, and determines and displays an input result according to the initial click position and the hand motion tracking data. The technical solution provided by the present application can improve the accuracy of gesture input in a virtual reality scenario.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and particularly to an interaction method, an electronic device, and an interaction system. Background Art

[0002] With the development of computer graphics technology and terminal technology, virtual reality (VR) technology, augmented reality (AR) technology, and mixed reality (MR) technology are increasingly applied to people's lives. In scenarios such as VR, AR, or MR, users can obtain an interactive immersive experience through a virtual fusion environment constructed by a head-mounted display (HMD) device (hereinafter referred to as a head-mounted device).

[0003] In a virtual environment, users can interact with the head-mounted device through a virtual user interface provided by the head-mounted device. For example, the head-mounted device can display a rendered virtual keyboard, and users can perform touch operations on the virtual keyboard to achieve gesture input. When perceiving a touch operation, the current mainstream solution is to use a camera on the head-mounted device to collect the user's hand image, identify and track the spatial positions of the key points of the user's hand based on consecutive multiple frames of hand images, and then determine the user's touch operation based on the tracked coordinate data. However, this method of realizing gesture input by perceiving the user's touch operation through a camera has a low accuracy rate of input recognition results due to reasons such as self-occlusion of the hand. Summary of the Invention

[0004] In view of this, this application provides an interaction method, an electronic device, and an interaction system for improving the accuracy rate of gesture input recognition results in a virtual reality scenario.

[0005] To achieve the above object, in a first aspect, an embodiment of this application provides an interaction method, which is applied to an interaction system. The interaction system includes a head-mounted device and a hand-worn device. The head-mounted device is equipped with a camera, and the hand-worn device is equipped with a motion sensor. The hand-worn device is also equipped with a vibration sensor and / or a pulse wave sensor. The method includes:

[0006] The head-mounted device displays a virtual keyboard;

[0007] The hand-worn device responds to a user's input trigger operation, and sends hand detection data to the head-mounted device according to the detected target sensor signal. The target sensor signal includes a bio-vibration wave signal detected by the vibration sensor and / or a pulse wave signal detected by the pulse wave sensor;

[0008] After the head-mounted device determines that an input trigger event has occurred based on the hand detection data, it determines the initial click position of the target finger relative to the virtual keyboard according to the hand image collected by the camera;

[0009] The hand-worn device sends the hand motion tracking data collected by the motion sensor to the head-mounted device;

[0010] In response to the user's input end operation, the head-mounted device determines and displays the input result according to the initial click position and the hand motion tracking data.

[0011] In the interaction method provided by the embodiments of the present application, the head-mounted device can display a virtual keyboard, and the hand-worn device can send hand detection data to the head-mounted device after detecting the target sensor signal; the head-mounted device can determine an input trigger event based on the hand detection data to implement the detection of click actions. The hand-worn device can also transmit the hand motion tracking data of the target finger collected by the motion sensor to the head-mounted device. The head-mounted device can collect the user's hand image through the camera and determine and display the input result based on the hand image and the hand motion tracking data collected by the motion sensor. In the above solution, by using the signals detected by the vibration sensor or the pulse wave sensor to detect click actions and determining the input result based on the hand motion tracking data collected by the motion sensor, the problem of hand occlusion can be avoided, and thus the accuracy of the determined input result can be improved; in addition, the sampling frequency of the motion sensor is usually much higher than the frame rate of the camera, and the complexity of the signal processing algorithm is relatively low. Therefore, using the signal collected by the motion sensor to determine the input result can also reduce the output delay and improve the real-time experience of the user's gesture input.

[0012] In a possible implementation manner of the first aspect, the hand image includes the hand image tracking data collected by the camera. Determining and displaying the input result according to the initial click position and the hand motion tracking data includes:

[0013] Perform data fusion on the hand image tracking data and the hand motion tracking data to obtain the motion tracking data of the target finger;

[0014] Determine the input result according to the initial click position, the motion tracking data, the keyboard layout of the virtual keyboard, and the preset language model;

[0015] Display the input result.

[0016] In the above implementation manner, the head-mounted device can further improve the accuracy of the determined input result by performing data fusion on the hand image tracking data collected by the camera and the hand motion tracking data collected by the IMU, and then determining the input result based on the motion tracking data of the target finger obtained by the fusion.

[0017] In a possible implementation of the first aspect, the hand image tracking data includes the image tracking data of the target part on the target finger or the image tracking data of the hand-worn device. In this way, the head-mounted device can extract the image tracking data of the target part or the hand-worn device from the hand image tracking data to determine the input result, so as to improve the processing efficiency.

[0018] In a possible implementation of the first aspect, visual markers are provided on the hand-worn device, and the hand image tracking data includes the image tracking data of the visual markers. In this way, the head-mounted device can extract the image tracking data of the visual markers from the hand image tracking data to determine the input result, so as to improve the processing efficiency.

[0019] In a possible implementation of the first aspect, when the head-mounted device determines that the target sensor signal meets the preset requirements according to the hand detection data, it determines that an input trigger event has occurred. This can improve the accuracy of gesture input trigger detection.

[0020] In a possible implementation of the first aspect, the hand detection data includes indication information. Sending the hand detection data to the head-mounted device according to the detected target sensor signal includes:

[0021] Performing feature extraction on the detected target sensor signal;

[0022] When the extracted target signal features meet the preset requirements, sending the indication information to the head-mounted device;

[0023] Correspondingly, when the head-mounted device receives the indication information, it determines that the target sensor signal meets the preset requirements.

[0024] In the above implementation, by detecting whether the target sensor signal meets the preset requirements on the hand-worn device, the data transmission volume between the hand-worn device and the head-mounted device can be reduced, thereby improving the transmission efficiency and saving the transmission energy consumption of the hand-worn device at the same time.

[0025] In a possible implementation of the first aspect, when the head-mounted device determines that the input trigger operation acts on the virtual keyboard according to the hand image collected by the camera, it determines that an input trigger event has occurred. This can facilitate the subsequent recognition of the input result and save energy consumption.

[0026] In a possible implementation of the first aspect, when the head-mounted device determines that the hand wearing the hand-worn device is in a target gesture according to the hand image collected by the camera, it determines that an input trigger event has occurred. This can improve the accuracy of gesture input trigger detection.

[0027] In a possible implementation of the first aspect, the rendering position of the virtual keyboard is on the physical plane, the input trigger operation is the operation of the target finger clicking on the virtual keyboard on the physical plane, and the input end operation is the operation of the target finger leaving the virtual keyboard on the physical plane.

[0028] In a possible implementation of the first aspect, the input trigger operation is the pinching operation of the target finger and the thumb, and the input end operation is the releasing operation of the target finger and the thumb.

[0029] In a possible implementation of the first aspect, the rendering position of the virtual keyboard is in the air. This can improve the convenience of user operation.

[0030] In a possible implementation of the first aspect, the hand-worn device responds to the user's sliding input operation and sends the hand movement tracking data collected by the motion sensor to the head-mounted device. In this way, the user can input content through the sliding input operation, thereby improving the flexibility of the user's input operation.

[0031] In a possible implementation of the first aspect, the hand-worn device is a finger-worn device, and the target finger is the finger wearing the hand-worn device. This can facilitate user wearing and better detect the movement of the target finger, thereby improving the accuracy of the detection result.

[0032] In a possible implementation of the first aspect, the vibration sensor and the motion sensor are the same IMU, and the head-mounted device and the hand-worn device communicate through Bluetooth connection.

[0033] In the above implementation, the vibration sensor and the motion sensor use the same IMU, which can reduce the structural complexity of the hand-worn device and improve the accuracy of the detected hand movement tracking data; the head-mounted device and the hand-worn device communicate through Bluetooth connection, which can reduce power consumption and cost.

[0034] In a second aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, the memory is used to store a computer program; the processor is used to execute the method performed by the head-mounted device or the hand-worn device in the first aspect when calling the computer program.

[0035] In a third aspect, an embodiment of the present application provides an interaction system, characterized by including: a head-mounted device and a hand-worn device, the head-mounted device has a camera, the hand-worn device has a motion sensor, and the hand-worn device also has a vibration sensor and / or a pulse wave sensor. Among them, the head-mounted device is used to execute the method performed by the head-mounted device in the first aspect, and the hand-worn device is used to execute the method performed by the hand-worn device in the first aspect.

[0036] Fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method executed by the head-mounted device or the hand-worn device in the first aspect is implemented.

[0037] Fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to execute the method executed by the head-mounted device or the hand-worn device in the first aspect.

[0038] Sixth aspect, an embodiment of the present application provides a chip system, including a processor. The processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the method executed by the head-mounted device or the hand-worn device in the first aspect. Wherein, the chip system may be a single chip or a chip module composed of multiple chips.

[0039] It can be understood that for the beneficial effects of the second aspect to the sixth aspect, reference may be made to the relevant descriptions in the first aspect, which will not be elaborated herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the system architecture of the interaction system provided by an embodiment of the present application;

[0041] Figure 2 It is a schematic diagram of the functional structure of the head-mounted device provided by an embodiment of the present application;

[0042] Figure 3 It is a schematic diagram of the functional structure of the hand-worn device provided by an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of the flow of the interaction method provided by an embodiment of the present application;

[0044] Figure 5 It is a schematic diagram of the bio-vibration wave signal provided by an embodiment of the present application;

[0045] Figure 6 It is a schematic diagram of an input result provided by an embodiment of the present application;

[0046] Figure 7 It is a schematic diagram of another input result provided by an embodiment of the present application;

[0047] Figure 8 It is a schematic diagram of the structure of an interaction device provided by an embodiment of the present application;

[0048] Figure 9 It is a schematic diagram of the structure of another interaction device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. The terms used in the implementation part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0050] To facilitate the understanding of the technical solutions in the embodiments of the present application, some terms involved in the embodiments of the present application will be explained below:

[0051] VR: Virtual Reality technology, which is a computer simulation system that can create and experience virtual worlds. This technology uses a computer to generate a simulated environment, enabling users to immerse themselves in the virtual environment, generating a three-dimensional visual, auditory, tactile, and even olfactory sense, and enabling real-time interaction through language, gestures, etc.

[0052] AR: Augmented Reality technology, which is a technology that combines virtual information with the real world. By using various technical means such as multimedia technology, three-dimensional modeling technology, intelligent interaction technology, and sensor technology, information such as text, images, three-dimensional models, music, and videos generated by a computer is superimposed on the real world, enabling the digital and physical information to complement each other, thereby realizing the "augmentation" of the real world.

[0053] MR: Mixed Reality technology, which is a further development of VR and AR technologies. By introducing real-world scene information into the virtual environment, it builds a bridge for interactive feedback information between the virtual world, the real world, and the user, thereby enhancing the realism of the user experience. Mixed reality generally uses optical see-through technology to superimpose virtual images on the human eyeball.

[0054] Technologies related to virtual environments such as VR, AR, and MR can all be generally referred to as virtual reality technologies in a broad sense. In some descriptions of the present application, virtual reality technology can be understood in a broad sense.

[0055] The technical solutions of the present application can be applied to VR devices, AR devices, or MR devices, such as common head-mounted display devices, etc. In the embodiments of the present application, the head-mounted display device is taken as an example for illustrative purposes.

[0056] In scenarios such as VR, AR, or MR, digital virtual scenes are involved. Virtual scenes are usually realized through VR devices, AR devices, or MR devices. To provide users with an interactive immersive experience, physical hardware such as joysticks and touchpads, or methods such as voice commands and gesture operations can be used for users to input.

[0057] In the specific implementation process of input through gesture operations, the gesture operations of the user can be obtained through the sensor system of the device. For example, the hand images of the user during the operation of the virtual keyboard can be collected through the camera on the head-mounted device, and the spatial positions of the key points of the user's hand are segmented, recognized, and tracked based on the collected hand images, and then the touch operations (i.e., gesture operations) of the user on the virtual keyboard are determined based on the tracking data. However, in this way of realizing gesture input by sensing the user's touch operations through the camera, due to the relative positional relationship between the camera on the head-mounted device and the user's hand, the clicking action of the finger is easily blocked by other parts of the hand, such as the back of the hand, resulting in the clicking action being easily misrecognized or missed; moreover, the algorithm for sensing the user's touch operations based on the camera is usually relatively complex, has high requirements for computing power, and the large time delay cannot meet the requirements for detecting fast clicking actions.

[0058] Therefore, the embodiments of the present application provide an interaction method, which realizes the detection of the clicking action by using the bio-vibration signals transmitted along the finger cortex and fat layer collected by the vibration sensor, and improves the accuracy and real-time experience of gesture input in the virtual reality scenario by combining the hand movement tracking data provided by the motion sensor and the hand images collected by the camera.

[0059] First, the system involved in the embodiments of the present application will be described below.

[0060] Figure 1 The system architecture diagram of the interaction system provided by the embodiments of the present application is shown in Figure 1 As shown, the interaction system may include: a head-mounted device 100 and a hand-worn device 200.

[0061] Among them, the head-mounted device 100 may be a helmet, glasses, etc., and the hand-worn device 200 may be a finger-worn device such as a ring or a finger sleeve, or an arm-worn device such as a bracelet or an armband. Figure 1 In this example, a finger-worn device is taken as an example for illustrative purposes.

[0062] A short - range communication connection can be established between the head - mounted device 100 and the hand - worn device 200, and data interaction is carried out through the established short - range communication connection. The technologies for establishing short - range communication connections include but are not limited to: wireless local area networks (WLAN) (such as wireless fidelity (Wi - Fi) networks), Bluetooth (BT) technology, ultra - wide band (UWB) technology, near - field communication (NFC) technology, infrared (IR) technology, general 2.4G / 5G band wireless communication technology, etc. In an embodiment of the present application, the Bluetooth technology is used to establish a short - range communication connection between the head - mounted device 100 and the hand - worn device 200 to reduce power consumption and development implementation costs. In the embodiments of the present application, the Bluetooth connection is taken as an example for illustrative description hereinafter.

[0063] The head - mounted device 100 can provide a virtual environment for the user to interact in ways such as VR, AR, or MR. The user can achieve user input through methods such as voice input or gesture input. Among them, gesture input can include air gesture input and interface - based gesture input (such as gesture input based on a virtual keyboard). For gesture input based on a virtual keyboard, after the head - mounted device 100 and the hand - worn device 200 establish a short - range communication connection, the head - mounted device 100 can obtain the hand movement tracking data collected by the hand - worn device 200 and identify gesture operations based on the obtained hand movement tracking data.

[0064] Figure 2 This is a schematic functional structure diagram of the head - mounted device 100 provided in the embodiment of the present application, as Figure 2 shown, the head - mounted device 100 may include: a processor 110, a memory 120, a communication module 130, a display screen 141, a camera 142, an audio module 143, a speaker 143A, a microphone 143B, a Universal Serial Bus (USB) interface 150, a charging management module 160, a power management module 161, a battery 162, a sensor module 170, a button 180, an indicator 190, etc.

[0065] It can be understood that the structure schematically shown in the embodiment of the present application does not constitute a specific limitation on the head - mounted device 100. In other embodiments of the present application, the head - mounted device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware.

[0066] The processor 110 may include one or more processing units. For example, the processor 110 may include an AP, a modem processor, a GPU, an ISP, a controller, a memory, a video codec, a DSP, a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0067] Among them, the controller may be the nerve center and command center of the head-mounted display device 100. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0068] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0069] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0070] The memory 120 can be used to store computer-executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the head-mounted device 100 by running the instructions stored in the memory 120. The memory 120 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc. The data storage area can store data created during the use of the head-mounted device 100 (such as image data, etc.). In addition, the memory 120 can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a Universal Flash Storage (UFS), etc.

[0071] The communication module 130 can provide solutions for wireless communications including WLAN (such as Wi-Fi network), Bluetooth, global navigation satellite system (GNSS), frequency modulation (FM), NFC, IR, etc. applied on the head-mounted device 100. The communication module 130 can be one or more devices integrating at least one communication processing module.

[0072] The display screen 141 can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a Micro LED, a quantum dot light emitting diode (QLED), etc.

[0073] In some embodiments, the head-mounted device 100 can have a transparent or semi-transparent display screen 142, and the user can directly view the physical environment through the display screen 142; the head-mounted device 100 can present a virtual environment on the transparent or semi-transparent display screen 142. The head-mounted device 100 can also have an opaque display screen 142, and the head-mounted device 100 can capture images or videos of the physical environment, and combine the captured images or videos with the virtual environment and present them on the opaque display screen 142.

[0074] The camera 142 is used to capture static images (such as the physical environment around the device) or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a Charge Coupled Device (CCD) or a Complementary Metal-Oxide-Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard formats such as RGB and YUV.

[0075] In some embodiments, the head-mounted display device 100 may include one or more cameras 142. Each camera 142 can be a traditional grayscale, RGB monocular camera, or a depth camera (for example, a camera that realizes depth imaging based on binocular disparity, a camera that realizes depth imaging based on time of flight). The head-mounted display device 100 can capture images or videos of the physical environment through one or more cameras 142 to superimpose and display the physical environment in the virtual environment, or perceive information such as the position of physical objects in the physical environment based on the captured images or videos, and superimpose and display virtual objects on the physical objects, such as displaying a virtual keyboard on a physical surface.

[0076] The head-mounted display device 100 can implement audio functions through the audio module 143, the speaker 143A, the microphone 143B, and the application processor, etc. For example, the head-mounted display device 100 can simulate the sounds in the physical environment for playback, and can also receive the user's voice information, etc.

[0077] The audio module 143 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into digital audio signals. The audio module 143 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 143 can be disposed in the processor 110, or some functional modules of the audio module 143 can be disposed in the processor 110.

[0078] The speaker 143A, also known as the "horn", is used to convert the audio electrical signal into a sound signal. The user can listen to audio information through the speaker 143A.

[0079] The microphone 143B, also known as a "microphone", is used to convert sound signals into electrical signals. When sending voice messages, the user can speak close to the microphone 143B with their mouth to input the sound signal into the microphone 143B. The head-mounted device 100 can be provided with at least one microphone 143B. In some other embodiments, the head-mounted device 100 can be provided with two microphones 143B, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the head-mounted device 100 can also be provided with three, four or more microphones 143B to collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0080] The USB interface 150 is an interface that conforms to the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 150 can be used to connect a charger to charge the head-mounted device 100, and can also be used to transfer data between the head-mounted device 100 and peripheral devices. This interface can also be used to connect other head-mounted devices 100.

[0081] The charging management module 160 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 160 can receive the charging input from a wired charger through the USB interface 150. In some embodiments of wireless charging, the charging management module 160 can receive the wireless charging input through the wireless charging coil of the head-mounted device 100. While charging the battery 162, the charging management module 160 can also supply power to the terminal device through the power management module 161.

[0082] The power management module 161 is used to connect the battery 162, the charging management module 160 and the processor 110. The power management module 161 receives the input from the battery 162 and / or the charging management module 160 and supplies power to the processor 110, the memory 120, the communication module 130, the camera 142, etc. The power management module 161 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 161 can also be provided in the processor 110. In some other embodiments, the power management module 161 and the charging management module 160 can also be provided in the same device.

[0083] The sensor module 170 can include a pressure sensor, a gyroscope sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, etc.

[0084] The button 180 includes a power-on button, volume buttons, etc. The button 180 can be a mechanical button or a touch button. The head-mounted device 100 can receive button inputs and generate key signal inputs related to the user settings and function controls of the head-mounted device 100.

[0085] The indicator 190 can be an indicator light, which can be used to indicate the charging status, power change, or messages.

[0086] Figure 3 It is a schematic functional structure diagram of the wearable device 200 provided by the embodiments of the present application. As Figure 3 shown, the wearable device 200 can include: a processor 210, a memory 220, a communication module 230, a vibration sensor 241, a pulse wave sensor 242, a motion sensor 243, a USB interface 250, a charging management module 260, a power management module 261, a battery 262, a button 270, an indicator 280, etc.

[0087] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the wearable device 200. In other embodiments of the present application, the wearable device 200 can include more or fewer components than those shown, or combine certain components, or split certain components, or have different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0088] The processor 210 can include one or more processing units. A memory can also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory.

[0089] The memory 220 can be used to store computer-executable program codes, and the executable program codes include instructions. The processor 210 executes various functional applications and data processing of the wearable device 200 by running the instructions stored in the memory 220.

[0090] The communication module 230 can provide solutions for wireless communications including WLAN (such as Wi-Fi networks), Bluetooth, NFC, IR, etc. applied on the wearable device 200. The communication module 230 can be one or more devices integrating at least one communication processing module.

[0091] The vibration sensor 241 can be a piezoelectric sensor, an optical sensor, a motion sensor, etc. that can detect bio-vibration wave signals.

[0092] The pulse wave sensor 242 is used to detect the pulse wave signal of the user's hand or arm, and it can be set on the side of the wearable device facing the user's skin. The pulse wave sensor 242 can specifically be a piezoelectric sensor, or a photoplethysmography (PPG) sensor, or other sensors capable of measuring the pulse wave signal. Among them, the pulse wave signal collected by the piezoelectric sensor is a pressure pulse wave signal, and the pulse wave signal collected by the photoplethysmography sensor is a photoplethysmograph (PPG) signal; the piezoelectric sensor can specifically be a piezoelectric thin film sensor or a piezoelectric ceramic sensor, etc., and the photoplethysmography sensor can specifically be implemented using a light emitting diode and a photodiode.

[0093] The motion sensor 243 can be a 3-axis accelerometer (ACC), a 3-axis gyroscope sensor, or a 6-axis / 9-axis inertial measurement unit (IMU). In an embodiment of the present application, the motion sensor 243 uses an IMU to improve the detection accuracy. The vibration sensor 241 can also use the same IMU to increase the structural complexity of the wearable device.

[0094] The USB interface 250 is an interface compliant with the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 250 can be used to connect a charger to charge the wearable device 200, and can also be used to transfer data between the wearable device 200 and peripheral devices. This interface can also be used to connect other wearable devices 200.

[0095] The charging management module 260 is used to receive the charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 260 can receive the charging input from the wired charger through the USB interface 250. In some embodiments of wireless charging, the charging management module 260 can receive the wireless charging input through the wireless charging coil of the wearable device 200. While charging the battery 262, the charging management module 260 can also supply power to the terminal device through the power management module 261.

[0096] The power management module 261 is used to connect the battery 262, the charging management module 260, and the processor 210. The power management module 261 receives the inputs from the battery 262 and / or the charging management module 260 and supplies power to the processor 210, the memory 220, the communication module 230, the camera 142, etc. The power management module 261 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 261 can also be disposed in the processor 210. In some other embodiments, the power management module 261 and the charging management module 260 can also be disposed in the same device.

[0097] The button 270 includes a power-on button, etc. The button 270 can be a mechanical button or a touch button. The wearable device 200 can receive button inputs and generate key signal inputs related to the user settings and function controls of the wearable device 200.

[0098] The indicator 280 can be an indicator light and can be used to indicate the charging status, the change in battery power, and can also be used to indicate messages, etc.

[0099] In this embodiment, the vibration sensor 241 on the wearable device 200 can detect the bio-vibration wave signals generated when the target finger performs actions such as tapping or pinching. The pulse wave sensor 242 can detect the pulse wave signals of the hand or arm when the target finger performs actions such as tapping or pinching. The IMU (i.e., the motion sensor 243) can detect the acceleration signals of the user's hand when the target finger performs actions such as tapping or pinching, and can also detect information such as the angular velocity and the geomagnetic field of the user's hand. Correspondingly, the IMU can include a three-axis accelerometer, or can also include a three-axis gyroscope and a three-axis magnetic sensor to respectively detect the acceleration, angular velocity, and geomagnetic field of the user's hand. Among them, when the wearable device 200 is a finger-worn device, the target finger can be the finger on which the user wears the finger-worn device, and the pulse wave sensor 242 can detect the pulse wave signal of the target finger; when the wearable device 200 is an arm-worn device, the target finger can be any finger of the hand wearing the wearable device 200, such as the index finger, and the pulse wave sensor 242 can detect the pulse wave signal of the user's arm.

[0100] The hand - worn device 200 can establish a short - range communication connection with the communication module 130 of the head - mounted device 100 through the communication module 230. After the connection is established, the hand - worn device 200 can send corresponding waveform data or features to the head - mounted device 100 after the pulse - wave sensor and / or vibration sensor detect relevant signals (hereinafter referred to as target sensor signals); the head - mounted device 100 can determine an input trigger event based on the waveform data or features to implement the detection of click actions. The hand - worn device 200 can also transmit hand motion tracking data collected by the IMU, such as linear acceleration, angular velocity, etc., to the head - mounted device 100. The head - mounted device 100 can simultaneously collect hand image tracking data of the user through the camera, fuse the hand motion tracking data collected by the IMU and the hand image tracking data collected by the camera, and then determine and display the input result. Among them, the head - mounted device 100 uses the signals detected by the pulse - wave sensor and / or vibration sensor to detect click actions, which can avoid the problem of hand occlusion when the user's touch operation is sensed by the camera, thus improving the accuracy of the input recognition result; moreover, when the head - mounted device 100 determines the input result, it fuses the hand motion tracking data collected by the IMU and the hand image tracking data collected by the camera for determination, which can also overcome the drift caused by the cumulative error of the IMU and improve the robustness of the tracking accuracy, further improving the accuracy of the determined input recognition result; in addition, the sampling frequency of the IMU is usually much higher than the frame rate of the camera, and the complexity of the signal - processing algorithm is relatively low. Therefore, the head - mounted device 100 uses the signals collected by the IMU to determine the input result, which can also reduce the output delay and enhance the real - time experience of the user's gesture input.

[0101] In some embodiments, the sampling frequency of the IMU can be higher than 100 Hz to improve the recognition accuracy and reduce the output delay to meet the requirements of detecting fast click actions.

[0102] For each input operation of the user, the hand - worn device 200 and the head - mounted device 100 can determine the input result through the above - mentioned interaction process. The input process corresponding to one input operation of the user is described below.

[0103] Figure 4 It is a schematic flowchart of the interaction method provided by the embodiment of the present application. As Figure 4 shown, the method may include the following steps:

[0104] S110. The head - mounted device displays a virtual keyboard.

[0105] As mentioned above, the head - mounted device can provide a virtual scene for the user to interact. The virtual scene may include content such as a virtual keyboard as an interaction interface for the user to complete input based on gestures.

[0106] Specifically, the head-mounted device can use 3D reconstruction algorithms such as the simultaneous localization and mapping (SLAM) algorithm or the Structure From Motion (SFM) algorithm based on monocular vision to display the virtual keyboard. This embodiment does not make special limitations on the specific 3D reconstruction algorithm used.

[0107] The rendering position of the virtual keyboard is on a physical plane. For example, it can be on a desktop or a wall; it can also be in the air. For example, the virtual keyboard is on a spatial plane 30 centimeters in front of the user's field of view.

[0108] Among them, the virtual keyboard can be a virtual keyboard with a QWERTY layout, or a virtual keyboard with other layouts. For example, a portable terminal keyboard.

[0109] S120. In response to the user's input trigger operation, the hand-worn device sends hand detection data to the head-mounted device according to the detected target sensor signal.

[0110] Specifically, the user can perform input operations on the virtual keyboard. The input operation can be a click operation or a swipe operation.

[0111] Among them, when the user performs an input operation, it can be achieved with a single finger or multiple fingers. For example, when the virtual keyboard is in the virtual space, the user can use a two-finger pinch operation to achieve click or swipe input, which can enable the user to feel a certain tactile feedback and improve the user's input experience to a certain extent. Among the input operations achieved with multiple fingers, the two-finger pinch operation is more convenient for the user to operate. In this embodiment, the two-finger pinch operation will be used as an example for illustrative description later.

[0112] For a single-finger input operation, the click operation can be a single click operation of the target finger on the physical plane where the virtual keyboard is located, and the swipe operation can be an operation where the target finger clicks on a certain key on the physical plane and then swipes to other keys and then lifts. The starting operation (i.e., the input trigger operation) of a single input operation is the operation of the target finger clicking on the virtual keyboard on the physical plane, and the input end operation is the operation of the target finger leaving the virtual keyboard on the physical plane.

[0113] For a single input operation implemented by multiple fingers, a click operation can be a pinching operation performed by the target finger and other fingers (such as the thumb) on a key on the virtual keyboard. A sliding operation can be an operation where the target finger and other fingers pinch together on a certain key on the virtual keyboard and then slide to other keys and release. That is, the pinching operation can be considered as a click operation on the key corresponding to the pinching position. The input trigger operation for a single input operation is the pinching operation of the target finger and the thumb, and the input end operation is the releasing operation of the target finger and the thumb.

[0114] When the user performs an input trigger operation by clicking on the virtual keyboard on the physical plane or pinching two fingers on the virtual keyboard, the target finger will vibrate. Correspondingly, the vibration sensor on the hand-worn device can detect the bio-vibration wave signal. In addition, when the user performs the above input trigger operation, the blood flow will also change. Correspondingly, the pulse wave signal detected by the pulse wave sensor on the hand-worn device will change.

[0115] The hand-worn device can send the target sensor signal detected by the vibration sensor and / or the pulse wave sensor to the head-mounted device as hand detection data for the head-mounted device to determine the input trigger event and trigger the gesture input detection process. Among them, using a single target sensor signal to determine the input trigger event can reduce the algorithm complexity, and only one of the sensors needs to be set on the hand-worn device, so the structural complexity of the hand-worn device can also be reduced; using the target sensor signals detected by both the vibration sensor and the pulse wave sensor to determine the input trigger event can improve the accuracy of the detection result. In specific implementation, it can be selected according to needs whether to use a single sensor or multiple sensors, and this embodiment does not make special limitations on this.

[0116] Considering that the user's hand may also vibrate when it is slightly bumped, in order to improve the accuracy of gesture input trigger detection, in this embodiment, the gesture input detection process of the head-mounted device can be triggered when the above target sensor signal meets the preset requirements.

[0117] Among them, the process of determining whether the target sensor signal meets the preset requirements can be executed on the head-mounted device or on the hand-worn device. If this determination process is executed on the head-mounted device, the hand-worn device only needs to continuously send the original data to the head-mounted device, which can reduce the device complexity of the hand-worn device to a certain extent; if this determination process is executed on the hand-worn device, when the hand-worn device sends the hand detection data, it can send the determination result only when the target sensor signal meets the preset requirements, which can reduce the data transmission volume between the hand-worn device and the head-mounted device, thus improving the transmission efficiency, and at the same time, it can also save the transmission power consumption of the hand-worn device. The execution entity of the above determination process can be specifically selected according to needs, and this embodiment does not make special limitations on this. In this embodiment, an example is given with the hand-worn device executing this determination process for illustrative purposes.

[0118] When making a specific determination, the hand-worn device can extract features from the target sensor signal detected by the vibration sensor and / or the pulse wave sensor, and when the extracted target signal features meet the preset requirements, send the indication information as the hand detection data to the head-mounted device.

[0119] Taking the vibration sensor as the IMU as an example, Figure 5 is a schematic diagram of the bio-vibration wave signal provided by the embodiment of the present application. In this figure, when the target finger clicks twice on the virtual keyboard on the desktop, the waveform diagrams corresponding to the x-axis, y-axis, and z-axis of the 3-axis accelerometer in the IMU are shown. Among them, the abscissa in each waveform diagram represents time or sampling points, and the ordinate represents acceleration or an electrical signal that can indicate acceleration. Figure 5 In, an example is given with the abscissa representing time and the ordinate representing acceleration for illustrative purposes.

[0120] As Figure 5 shown, when the target finger generates vibration, the waveform of the acceleration signal detected by the IMU will change, and the hand-worn device can periodically detect whether the waveform characteristics of the acceleration signal meet the preset requirements with a preset time window (such as 1 second).

[0121] Specifically, the waveform characteristics of the acceleration signal can include time-domain characteristics (such as the amplitude, mean, and variance of the x-axis, y-axis, and z-axis) and transform-domain characteristics (such as frequency domain, gradient domain, wavelet domain characteristics, etc.). When performing feature extraction, multiple features among these features (i.e., target signal features) can be selected, and then relevant feature extraction algorithms are used for feature extraction. The specific feature extraction algorithm is not particularly limited in this embodiment.

[0122] Similar to the acceleration signal, the waveform characteristics of the pulse wave signal can also include time-domain characteristics and transform-domain characteristics. Some target signal features can be selected and relevant feature extraction algorithms are used for feature extraction. The specific feature extraction algorithm is also not particularly limited in this embodiment.

[0123] After the feature extraction is completed, it can be determined whether the extracted target signal features meet the preset requirements. Among them, for each target signal feature, a corresponding threshold can be set. When each target signal feature meets the corresponding threshold requirement, it can be considered that the target signal feature meets the preset requirements; alternatively, the number of target signal features that meet the threshold requirement can also be counted. When this number meets the preset number, it is considered that the target signal feature meets the preset requirements. Here, only two judgment methods are exemplified, which are not used to limit the present application. In specific implementation, other judgment methods can be adopted according to needs, and the present application does not make special limitations in this regard. In addition, if a vibration sensor uses other sensors such as a piezoelectric sensor or an optical sensor, it can also be determined whether the bio-vibration wave signal meets the preset requirements based on waveform features. The specific judgment method is similar to the above-mentioned judgment method and will not be elaborated here.

[0124] If the target signal features meet the preset requirements, the hand-worn device can send corresponding indication information to the head-mounted device. Among them, this indication information can be used as hand detection data and sent to the head-mounted device through a Bluetooth data packet.

[0125] Specifically, a target field can be added to the Bluetooth data packet, and this indication information can be carried in this target field. For example, a target field can be added to the protocol data unit (PDU) field of the Bluetooth data packet, and "1" is used to represent this indication information in this target field. Correspondingly, after receiving the data packet, the head-mounted device can consider that the bio-vibration wave signal meets the preset requirements when it determines that the value of this target field is 1.

[0126] To improve the reliability of data transmission, the hand-worn device can perform error correction coding on the hand detection data and then add it to the Bluetooth data packet. Among them, error correction coding techniques can adopt parity check coding, convolutional coding, reed-solomon (RS) coding, cyclical redundancy check (CRC) coding, etc.

[0127] S130. After the head-mounted device determines that an input trigger event has occurred based on the hand detection data, it determines the initial click position of the target finger relative to the virtual keyboard according to the hand image collected by the camera.

[0128] After the head-mounted device receives a Bluetooth data packet, it can perform error correction verification and decoding on the data carried in the data packet to obtain hand detection data (such as the above-mentioned indication information). When it is determined that the above-mentioned target sensor signal meets the preset requirements based on the hand detection data, it is determined that an input trigger event has occurred. At this time, the gesture input detection process can be triggered, and a hand image is collected through a camera to determine the input result. This can save some processing energy consumption. Moreover, in the case where the camera for collecting the physical environment is different from the camera for collecting the hand image, the head-mounted device can start the camera for collecting the hand image when it is determined that the target sensor signal meets the preset requirements, which can also save energy consumption.

[0129] Considering that the user's input trigger operation may be outside the virtual keyboard area, in this embodiment, the head-mounted device can further determine whether the user's input trigger operation acts on the virtual keyboard when receiving the hand detection data. If so, the gesture input detection process is triggered, which can facilitate the recognition of subsequent input results and also save energy consumption.

[0130] Specifically, the head-mounted device can determine whether the fingertip of the target finger points to the area within the virtual keyboard according to the hand image collected by the camera. If so, it can be considered that the user's input trigger operation acts on the virtual keyboard.

[0131] As mentioned above, the user's hand may also vibrate when it is slightly bumped. To further improve the accuracy of gesture input trigger detection, in this embodiment, the head-mounted device can also determine whether the hand wearing the hand-mounted device is a target gesture according to the hand image collected by the camera. When it is determined that the hand wearing the hand-mounted device is a target gesture, the gesture input detection process of the head-mounted device is triggered to determine the subsequent input result.

[0132] Among them, the target gesture can be, for example, a pointing gesture with the index finger extended and other fingers bent, or a gesture of pinching the index finger and thumb. Specifically, the target gesture can be set according to needs, and this embodiment does not make special limitations on this. To facilitate user use, the head-mounted device can also provide a gesture customization function for the user to customize the target gesture.

[0133] If the gesture input detection process is triggered, the head-mounted device can further identify the initial click position of the target finger relative to the virtual keyboard from the hand image collected by the camera. Specifically, various relevant image recognition algorithms can be used, and this embodiment does not make special limitations on this.

[0134] S140. The hand-mounted device sends the hand motion tracking data collected by the motion sensor to the head-mounted device.

[0135] After the hand - worn device sends the hand detection data to the head - mounted device, it can continue to send the hand motion tracking data collected by the motion sensor to the head - mounted device so that the head - mounted device can identify the input result. Among them, the hand motion tracking data can include a series of local coordinates collected by the motion sensor.

[0136] S150. In response to the user's input end operation, the head - mounted device determines and displays the input result according to the initial click position and the hand motion tracking data.

[0137] In the case where the hand - worn device is a finger - worn device, the head - mounted device can identify whether the user has performed an input end operation based on the received hand motion tracking data, that is, identify whether the user has ended the current input operation. If so, it can continue to determine the input result.

[0138] Specifically, as described above, the input end operation can be the operation of the target finger leaving the virtual keyboard on the physical plane, or the operation of the target finger releasing the thumb; it can be determined whether the user has ended the current input operation according to the motion trajectory of the target finger in the hand motion tracking data.

[0139] It can be understood that the process of determining whether the user has ended the current input operation can also be executed on the hand - worn device. Correspondingly, when the hand - worn device determines that the user has ended the current input operation, the hand motion tracking data sent to the head - mounted device can carry an indication information indicating the input end. The implementation method of this indication information is similar to the indication information indicating that the target signal feature meets the preset requirements described above, and can be represented by a preset field. For example, the above - mentioned target field can be continued to be used. When the field value is 2, it indicates the input end. Further, when the hand - worn device determines that the user has ended the current input operation, it can pause sending the hand motion tracking data to the head - mounted device to save transmission energy consumption. In addition, as described above, the user's input operation can be a click operation or a slide operation. If it is a click operation, the hand motion tracking data is the indication information indicating the input end; if it is a slide operation, the hand motion tracking data can also include the data collected by the IMU during the sliding of the target finger.

[0140] When specifically determining the input result, if it is a click operation, after the head - mounted device determines that the user has ended the current input operation according to the hand motion tracking data, it can then determine the input result according to the initial click position and the keyboard layout of the virtual keyboard.

[0141] If it is a slide operation, after the head - mounted device determines that the user has ended the current input operation according to the hand motion tracking data, it determines the input result according to the initial click position, the hand motion tracking data, and the keyboard layout of the virtual keyboard.

[0142] When the hand-held device is an arm-worn device, for the input end operation where the target finger leaves the virtual keyboard on the physical plane, the head-mounted device can determine whether the user ends the current input operation according to the movement trajectory of the target finger in the hand movement tracking data. After determining that the user ends the current input operation, it continues to determine the input result. For the input end operation where the target finger and the thumb are released, the user can perform this operation with a certain release force. At this time, bio-vibration wave signals will also be generated in the user's hand. The head-mounted device can then determine whether the user ends the input operation based on the bio-vibration wave signals detected by the hand-held device. The specific detection process is similar to the detection process of the input trigger operation and will not be elaborated here. Among them, during the judgment process, the head-mounted device can distinguish whether it is an input trigger operation or an input end operation according to the number of detected bio-vibration wave signals and / or signal characteristics (such as amplitude). Similar to the input trigger operation, when the user performs the input end operation with a certain release force, the blood flow in the user's hand will also change. The head-mounted device can also determine whether the user ends the input operation based on the pulse wave signals detected by the hand-held device. The specific judgment process is similar to the judgment process corresponding to the bio-vibration wave signals and will not be elaborated here.

[0143] To make it more convenient for users to use, for the input end operation where the target finger and the thumb are released, the head-mounted device can also determine whether the user ends the current input operation based on the hand image collected by the camera. Specifically, the head-mounted device can identify whether the target finger and the thumb are in a released gesture based on the hand image, and determine that the user ends the current input operation when it identifies that the target finger and the thumb are released, and then determine the input result.

[0144] To improve the accuracy of the determined input result, in this embodiment, in step S130, after the head-mounted device receives the hand detection data, when it determines that the user's input trigger operation acts on the virtual keyboard, it can collect the hand image tracking data of the user through the camera, that is, the hand image collected by the camera can include this hand image tracking data; correspondingly, when determining the input result, the head-mounted device can perform data fusion on the hand image tracking data collected by the camera and the hand movement tracking data collected by the IMU, and then based on the movement tracking data of the target finger obtained by the fusion, combined with the initial click position and the keyboard layout of the virtual keyboard, determine the input result.

[0145] Among them, the hand image tracking data includes the image tracking data of the target part on the target finger (such as finger key points like the fingertip or finger joint), and the image tracking data of the hand - worn device. The head - mounted device can extract the image tracking data of the target part or the hand - worn device from the hand image tracking data to determine the input result, so as to improve the processing efficiency. In addition, visual markers (such as LED lights, two - dimensional codes, or patterns, etc.) can be set on the hand - worn device. The image tracking data of the hand - worn device can include the image tracking data of the visual marker. The head - mounted device can extract the image tracking data of the visual marker from the hand image tracking data to determine the input result, so as to further improve the processing efficiency.

[0146] When performing data fusion, data fusion algorithms such as Kalman filtering or maximum likelihood estimation can be used, such as visual - inertial odometry algorithms like Kalman filtering under multi - state constraints, to fuse the hand image tracking data and the hand motion tracking data. For the specific data fusion algorithm used, this embodiment does not make a special limitation.

[0147] When specifically determining the input result, it can be determined in combination with a language model (such as an N - gram language model). Specifically, various relevant keyboard - based gesture input algorithms can be used. This embodiment does not make a special limitation on this.

[0148] Figure 6 This is a schematic diagram of an input result provided by an embodiment of the present application. This figure shows the input result of a single - click operation of the user. As Figure 6 shown, after the user enters the string "hello", and then clicks on positions 1A and 1B (represented by dashed circles in the figure) through two single - click operations in sequence. For the second single - click operation, the head - mounted device can determine various candidate input results: wo, world, and would based on the initial click position, the keyboard layout of the virtual keyboard, and the N - gram language model; the user can further select the final input result from these candidate input results.

[0149] Figure 7 This is another schematic diagram of an input result provided by an embodiment of the present application. This figure shows the input result of a sliding operation of the user. As Figure 7 shown, position 2A represents the start position of the sliding gesture (i.e., the initial click position), positions 2B and 2C represent the positions passed by the sliding gesture, and position 2D represents the end position of the sliding gesture. The head - mounted device can determine the motion trajectory and some motion characteristics of the target finger based on the motion tracking data of the target finger, and then combine the keyboard layout of the virtual keyboard and the N - Gram language model to determine various candidate input results: apple, Apple, and apples; the user can further select the final input result from the candidate input results.

[0150] Those skilled in the art can understand that the above embodiments are exemplary and are not used to limit the present application. Where possible, the execution order of one or several of the above steps can be adjusted, or selective combination can be performed to obtain one or more other embodiments. For example, there is no strict timing execution relationship between the process of determining the initial click position in step S130 and step S150 above, and the process of determining the initial click position can also be implemented during the process of the head-mounted device responding to the input end operation. Those skilled in the art can arbitrarily select and combine from the above steps as needed. All those that do not depart from the essence of the present application solution fall within the protection scope of the present application.

[0151] For the interaction method provided in this embodiment, the head-mounted device can display a virtual keyboard. After detecting the target sensor signal, the hand-worn device can send hand detection data to the head-mounted device. The head-mounted device can determine an input trigger event based on the hand detection data to implement the detection of click actions. The hand-worn device can also transmit the hand movement tracking data collected by the motion sensor to the head-mounted device. The head-mounted device can collect the user's hand image through the camera and determine and display the input result based on the hand image and the hand movement tracking data collected by the motion sensor. In the above solution, by using the signal detected by the vibration sensor or the pulse wave sensor to detect the click action and determining the input result based on the hand movement tracking data collected by the motion sensor, the problem of hand occlusion can be avoided, so the accuracy of the determined input result can be improved; in addition, the sampling frequency of the motion sensor is usually much higher than the frame rate of the camera, and the complexity of the signal processing algorithm is relatively low. Therefore, using the signal collected by the motion sensor to determine the input result can also reduce the output delay and improve the real-time experience of the user's gesture input.

[0152] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application provides an interaction device. This device embodiment corresponds to the foregoing method embodiment. For ease of reading, the details of the foregoing method embodiment will not be described one by one in this device embodiment. However, it should be clear that the device in this embodiment can correspondingly implement all the content of the foregoing method embodiment.

[0153] Figure 8 A schematic structural diagram of an interaction device provided in an embodiment of the present application. This device can be applied to a head-mounted device, such as Figure 8 As shown, the interaction device 300 provided in this embodiment can include: a display module 310, an input module 320, a processing module 330, and a communication module 340.

[0154] Among them, the display module 310 is used to support the head-mounted device to perform the display operation in step S110 in the above embodiment and / or other processes for the technologies described herein. The display module can be a touch screen or other hardware or a combination of hardware and software.

[0155] The input module 320 is used to receive input operations of the user on the head-mounted device, such as voice input, gesture input, etc., and / or other processes for the technologies described herein. The input module can be a touch screen or other hardware or a combination of hardware and software.

[0156] The processing module 330 is used to support the head-mounted device to perform the processing operation in step S130 in the above embodiment, S150 and / or other processes for the technologies described herein.

[0157] The communication module 340 is used to support the head-mounted device to perform the operation of receiving hand detection data in step S130 in the above embodiment and / or other processes for the technologies described herein.

[0158] The device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here.

[0159] Figure 9 As shown in the figure, it is a schematic structural diagram of another interaction device provided in an embodiment of the present application. This device can be applied to a hand-worn device, such as Figure 9 As shown, the interaction device 400 provided in this embodiment can include: a processing module 410 and a communication module 420.

[0160] Among them, the processing module 410 is used to support the hand-worn device to perform the processing operation in the above embodiment and / or other processes for the technologies described herein.

[0161] The communication module 420 is used to support the hand-worn device to perform steps S120 and S140 in the above embodiment and / or other processes for the technologies described herein.

[0162] The device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here.

[0163] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0164] The embodiment of this application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the foregoing method embodiment is implemented.

[0165] The embodiment of this application also provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to execute the method described in the foregoing method embodiment. Among them, the electronic device can be the above-mentioned head-mounted device or hand-worn device.

[0166] The embodiment of this application also provides a chip system, including a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the method described in the foregoing method embodiment. Among them, the chip system can be a single chip or a chip module composed of multiple chips.

[0167] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0168] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware with a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The foregoing storage medium can include various media that can store program codes, such as ROM or random access memory RAM, magnetic disks, or optical discs.

[0169] In the present application, the naming or numbering of steps does not mean that the steps in the method process must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can be changed in the order of execution according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0170] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0171] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the apparatus or unit can be in electrical, mechanical or other forms.

[0172] It should be understood that in the description of this application specification and the appended claims, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that comprises a series of steps or modules does not necessarily have to be limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or devices.

[0173] In the description of this application, unless otherwise specified, " / " means that the objects associated before and after are in an "or" relationship. For example, A / B can represent A or B. The "and / or" in this application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Among them, A and B can be singular or plural.

[0174] Also, in the description of this application, unless otherwise specified, "a plurality of" means two or more than two. "At least one of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c can be single or multiple.

[0175] As used in this application specification and the appended claims, the term "if" can be interpreted as "when...", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0176] In addition, in the description of the specification and the appended claims of the present application, the terms "first", "second", "third", etc. are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than that shown or described herein.

[0177] The reference to "one embodiment" or "some embodiments" or the like described in the specification of the present application means that a specific feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An interaction method, applied to an interaction system, characterized in that The interactive system includes a head-mounted device and a hand-worn device. The head-mounted device is equipped with a camera, and the hand-worn device is equipped with a motion sensor. The hand-worn device is also equipped with a vibration sensor and / or a pulse wave sensor. The method includes: The head-mounted device displays a virtual keyboard; In response to a user's input trigger operation, the hand-worn device sends hand detection data to the head-mounted device according to the detected target sensor signal. The target sensor signal includes a bio-vibration wave signal detected by the vibration sensor and / or a pulse wave signal detected by the pulse wave sensor; After the head-mounted device determines that an input trigger event has occurred based on the hand detection data, it determines an initial click position of the target finger relative to the virtual keyboard according to the hand image collected by the camera; The hand-worn device sends hand motion tracking data collected by the motion sensor to the head-mounted device; In response to the user's input end operation, the head-mounted device determines and displays an input result according to the initial click position and the hand motion tracking data.

2. The method according to claim 1, wherein The hand image includes hand image tracking data collected by the camera. Determining and displaying the input result according to the initial click position and the hand motion tracking data includes: Performing data fusion on the hand image tracking data and the hand motion tracking data to obtain motion tracking data of the target finger; Determining an input result according to the initial click position, the motion tracking data, the keyboard layout of the virtual keyboard, and a preset language model; Displaying the input result.

3. The method according to claim 2, wherein The hand image tracking data includes image tracking data of a target part on the target finger or image tracking data of the hand-worn device.

4. The method according to claim 3, wherein A visual marker is provided on the hand-worn device, and the hand image tracking data includes image tracking data of the visual marker.

5. The method according to claim 1, characterized in that When the head-mounted device determines that the target sensor signal meets a preset requirement based on the hand detection data, it determines that an input trigger event has occurred.

6. The method according to claim 5, characterized in that, The hand detection data includes indication information. Sending the hand detection data to the head-mounted device according to the detected target sensor signal includes: Performing feature extraction on the detected target sensor signal; When the extracted target signal feature meets a preset requirement, sending the indication information to the head-mounted device; Correspondingly, when the head-mounted device receives the indication information, it determines that the target sensor signal meets a preset requirement.

7. The method according to claim 1, characterized in that When the head-mounted device determines that the input trigger operation acts on the virtual keyboard according to the hand image collected by the camera, it determines that an input trigger event has occurred.

8. The method according to claim 1, wherein When the head-mounted device determines that the hand of the user wearing the hand-worn device is a target gesture according to the hand image collected by the camera, it determines that an input trigger event has occurred.

9. The method according to claim 1, wherein The rendering position of the virtual keyboard is on a physical plane. The input trigger operation is an operation in which the target finger clicks on the virtual keyboard on the physical plane, and the input end operation is an operation in which the target finger leaves the virtual keyboard on the physical plane.

10. The method according to claim 1, characterized in that The input trigger operation is a pinching operation of the target finger and the thumb, and the input end operation is an operation of releasing the target finger and the thumb.

11. The method according to claim 10, wherein, The rendering position of the virtual keyboard is in the air.

12. The method according to claim 1, wherein In response to the user's sliding input operation, the hand-worn device sends the hand movement tracking data collected by the motion sensor to the head-mounted device.

13. The method according to claim 1, wherein The hand-worn device is a finger-worn device, and the target finger is the finger wearing the hand-worn device.

14. The method according to any one of claims 1 to 13, characterized in that, The vibration sensor and the motion sensor are the same IMU, and the head-mounted device and the hand-worn device communicate through Bluetooth connection.

15. An electronic device, characterized in that, Comprising: A memory and a processor, the memory is used to store a computer program; the processor is used to execute the method performed by the head-mounted device or the hand-worn device in any one of claims 1-14 when calling the computer program.

16. An interaction system, characterized in that, Comprising: A head-mounted device and a hand-worn device, the head-mounted device has a camera, the hand-worn device has a motion sensor, and the hand-worn device also has a vibration sensor and / or a pulse wave sensor. Among them, the head-mounted device is used to execute the method performed by the head-mounted device in any one of claims 1-14, and the hand-worn device is used to execute the method performed by the hand-worn device in any one of claims 1-14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1-14.

Images