Information interaction method and head-mounted display device
By using identity recognizing and gesture classification models in the headset device, the application space and response functions are determined based on the user's gesture image, the problem of privacy protection in the headset device shared by multiple people is solved, and efficient user privacy protection is achieved.
Patent Information
- Application Number
- PCT/CN2024/079736
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2024-03-01
- Publication Date
- 2024-11-21
AI Technical Summary
When multiple family members share headset devices, how to effectively protect user privacy and prevent privacy leakage.
By obtaining the user's gesture image and using the identity re-identification model and gesture classification model, the user's application space and response functions are determined to protect user privacy. The method includes inputting the gesture image into the identity re-identification model to identify the user name and inputting the gesture image into the gesture classification model to determine the response function, ensuring that only authorized users can access a specific application space.
It effectively protects users' privacy, prevents unauthorized access, and improves the accuracy of user identity identification and the efficiency of determining response functions.
Smart Images

Figure CN2024079736_21112024_PF_FP_ABST
Abstract
Description
Information interaction method and head-mounted display device
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 15, 2023, with application number 202310548994.X and application name “A method for information interaction and a head-mounted display device”, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present invention relates to the field of virtual reality technology, and in particular to an information interaction method and a head-mounted display device. Background Art
[0004] Virtual reality (VR) and augmented reality (AR) are emerging multimedia technologies in recent years. VR technology presents virtual information to users without them being able to see the real world, completely detaching them from reality and placing them in a virtual world. AR technology overlays virtual information onto the user's field of view while allowing them to see the real world, blending the real and virtual worlds and allowing users to more intuitively experience the effects of augmented reality.
[0005] With the continuous development of computer software and hardware, extended reality (XR) devices that provide VR and / or AR technologies are becoming increasingly popular in families. When multiple family members use the same head-mounted display (HMD), the lack of privacy protection within the HMD is a pressing issue that needs to be addressed.
[0006] Summary of the Invention
[0007] The embodiments of the present application provide an information interaction method and a head-mounted display device for protecting user privacy in the head-mounted display device.
[0008] In a first aspect, embodiments of the present application provide an information interaction method, comprising: a head-mounted display device acquiring a first gesture image of a first user, and determining a first application space and a first response function corresponding to the first gesture image, wherein the first application space is the application space allocated by the head-mounted display device to the first user, and the first response function is the head-mounted display device's response to the first gesture image. In the first application space, the head-mounted display device executes the first response function. In this method, the first user can access application software in the first application space, and the first application space is accessible to the first user, effectively protecting the first user's privacy.
[0009] In one possible design, the head-mounted display device determines the first application space corresponding to the first gesture image, which may include: the head-mounted display device inputting the first gesture image into a first identity re-identification model to obtain a first username, and using the application space corresponding to the first username as the first application space, wherein the first identity re-identification model is trained using a first dataset, the first dataset including preset gesture images with preset username labels. With this design, the head-mounted display device inputs the first gesture image into the first identity re-identification model, and the first username can be determined based on the first gesture image, effectively improving recognition efficiency.
[0010] In one possible design, determining a first response function corresponding to a first gesture image by a head-mounted display device may include: inputting the first gesture image into a gesture classification model to obtain the first response function, wherein the gesture classification model is trained using a second dataset, the second dataset including preset gesture images with preset response function labels. With this design, the head-mounted display device inputs the first gesture image into the gesture classification model and determines the first response function based on the first gesture image, providing a convenient method for determining the first response function.
[0011] In one possible design, after the head-mounted display device obtains a first gesture image of a first user, the device may further include: obtaining a first iris image of the first user; the head-mounted display device determining a first application space corresponding to the first gesture image may include: the head-mounted display device inputting the first gesture image and the first iris image into a second identity re-identification model to obtain the first username, and using the application space corresponding to the first username as the first application space, wherein the second identity re-identification model is obtained by training a third dataset, the third dataset including a preset gesture image with a preset username label and a preset iris image with a preset username label. With this design, the head-mounted display device inputs the first gesture image and the first iris image into the second identity re-identification model, and determines the first username based on the first gesture image and the first iris image, which can effectively improve recognition accuracy.
[0012] In one possible design, the method further includes: the head-mounted display device adding the first hand gesture image labeled with the first user name to the first dataset, and using the updated first dataset to train the first identity re-identification model. With this design, the first identity re-identification model can still accurately identify the user even when the user's hand gestures are non-standard or when the user's hand texture information changes with age.
[0013] In one possible design, the head-mounted display device adds a first gesture image tagged with a first user name to a first data set. This may include: the head-mounted display device determines at least one preset gesture image tagged with the first user name from the first data set, then determines a first similarity between the at least one preset gesture image tagged with the first user name and the first gesture image; and then, if the first similarity is less than a first threshold and greater than a second threshold, the head-mounted display device adds the first gesture image tagged with the first user name to the first data set. This design provides a method for updating the first data set.
[0014] In one possible design, the method further includes: the head-mounted display device adding the first gesture image with the first response function label to a second dataset, and using the updated second dataset to train the gesture classification model. With this design, the gesture classification model can still accurately recognize gestures made by the user even when the user's gestures are non-standard or when the user's hand texture information changes with age.
[0015] In one possible design, the head-mounted display device adds a first gesture image with a first response function tag to a second dataset. This may include: the head-mounted display device determines at least one preset gesture image with the first response function tag from the second dataset, then determines a second similarity between the at least one preset gesture image with the first response function tag and the first gesture image; and then, if the second similarity is less than a third threshold and greater than a fourth threshold, the head-mounted display device adds the first gesture image with the first response function tag to the second dataset. This design provides a method for updating the second dataset.
[0016] In one possible design, the method further includes: the head-mounted display device adding the first gesture image labeled with the first username and the first iris image labeled with the first username to a third dataset, and using the updated third dataset to train the second identity re-identification model. With this design, the second identity re-identification model can still accurately identify users even when their gestures are non-standard, or when their hand texture information or iris changes with age.
[0017] In one possible design, the head-mounted display device adds the first gesture image tagged with the first username to the third dataset. This may include: the head-mounted display device determines at least one preset gesture image tagged with the first username from the third dataset, then determines a third similarity between the at least one preset gesture image tagged with the first username and the first gesture image; and then, if the third similarity is less than a fifth threshold and greater than a sixth threshold, the head-mounted display device adds the first gesture image tagged with the first username to the third dataset. This design provides a method for updating the third dataset.
[0018] In a second aspect, the present application further provides a head-mounted display device capable of implementing the method of the first aspect and any possible implementation thereof. The head-mounted display device may be implemented via hardware or via hardware executing corresponding software. The hardware or software may include one or more units or modules corresponding to the aforementioned functions.
[0019] In a third aspect, embodiments of the present application further provide a head-mounted display device, comprising modules / units for executing the methods of the first aspect and any possible implementation thereof. These modules / units may be implemented in hardware, or in hardware executing corresponding software implementations.
[0020] In a fourth aspect, an embodiment of the present application provides a head-mounted display device, comprising a memory and a processor; wherein the processor is configured to execute a computer program or instruction stored in the memory, so that the head-mounted display device implements the method in the above-mentioned first aspect and any possible implementation manner of the first aspect.
[0021] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes a computer program. When the computer program runs on a head-mounted display device, the computer-readable storage medium implements the method in the above-mentioned first aspect and any possible implementation method of the first aspect.
[0022] In a sixth aspect, an embodiment of the present application further provides a computer program product. When the computer program product is run on a head-mounted display device, the head-mounted display device implements the method in the above-mentioned first aspect and any possible implementation manner of the first aspect.
[0023] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] FIG1a is a schematic diagram of the hardware structure of a head-mounted display device provided in an embodiment of the present application;
[0025] FIG1b is a schematic diagram of the hardware structure of a head-mounted display device provided in an embodiment of the present application;
[0026] FIG1c is a schematic diagram of the software structure of a head-mounted display device provided in an embodiment of the present application;
[0027] FIG2 is a flow chart of an information interaction method provided in an embodiment of the present application;
[0028] FIG3 is a schematic structural diagram of a first user interface provided in an embodiment of the present application;
[0029] FIG4 is a schematic structural diagram of a first user interface provided in an embodiment of the present application;
[0030] FIG5 is a schematic structural diagram of a first user interface provided in an embodiment of the present application;
[0031] FIG6 is a schematic diagram of a preset gesture image provided by an embodiment of the present application;
[0032] FIG7 is a schematic diagram of a preset gesture image provided by an embodiment of the present application;
[0033] FIG8 is a schematic diagram of a preset gesture image provided by an embodiment of the present application;
[0034] FIG9 is a schematic diagram of the structure of a gesture joint point provided in an embodiment of the present application;
[0035] FIG10 is a schematic diagram of the structure of a call interface provided in an embodiment of the present application;
[0036] FIG11 is a schematic diagram of a preset gesture image provided by an embodiment of the present application;
[0037] FIG12 is a schematic diagram of a preset gesture image provided by an embodiment of the present application;
[0038] FIG13 is a flow chart of an information interaction method provided in an embodiment of the present application;
[0039] FIG14 is a schematic diagram of a first gesture image provided in an embodiment of the present application;
[0040] FIG15 is a schematic diagram of the hardware structure of a head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0042] Below, some terms used in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.
[0043] XR devices can be VR devices or AR devices. VR devices can be position trackers, data gloves, head-mounted devices, etc., while AR devices can be mobile phones, tablets, laptops, or wearable devices with wireless communication capabilities (such as smart watches or smart glasses).
[0044] The XR device includes a device capable of performing data processing functions (such as a processor, an application processor, an image processor, or other processors). Exemplary embodiments of the XR device include but are not limited to a device equipped with Or devices with other operating systems. The above-mentioned XR device can also be other portable devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel).
[0045] In conjunction with the accompanying drawings, taking the XR device as a head-mounted display device (hereinafter referred to as a head-mounted display device) as an example, the structure of the above-mentioned XR device is further explained.
[0046] FIG1a illustrates a schematic diagram of the structure of a head-mounted display device 100 and a schematic diagram of the structure of a handle 101 provided in an embodiment of the present application. Based on the head-mounted display device 100 shown in FIG1a , the head-mounted display device may also have other structural variations, which are not limited herein. The head-mounted display device 100 may be controlled by a user via the handle 101 shown in FIG1a , directly with a user's fingertips, or via other devices, which are not limited herein.
[0047] A schematic diagram of the hardware structure corresponding to the head-mounted display device 100 is shown in FIG1b . The head-mounted display device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0048] It should be understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the head-mounted display 100. In other embodiments of this application, the head-mounted display device 100 may include more or fewer components than those shown in FIG1b , or may combine or separate certain components, or may have different component arrangements. The components shown in FIG1b may be implemented in hardware, software, or a combination of software and hardware.
[0049] The following is a detailed introduction to the components of the head display device 100 shown in Figure 1b.
[0050] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. The controller may be the nerve center and command center of the head-mounted display device 100. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.
[0051] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory, thereby avoiding repeated accesses and reducing processor 110 latency, thereby improving system efficiency.
[0052] In some embodiments, the processor 110 may include one or more interfaces. For example, the interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.
[0053] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C busses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc. via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby enabling the touch function of the head-mounted display device 100.
[0054] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.
[0055] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0056] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.
[0057] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the head-mounted display device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the head-mounted display device 100.
[0058] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0059] The USB interface 130 complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the head-mounted display device 100, or to transfer data between the head-mounted display device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as mobile phones.
[0060] It should be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely illustrative and does not constitute a structural limitation on the head-mounted display device 100. In other embodiments of the present application, the head-mounted display device 100 may also adopt a different interface connection method from the above embodiments, or a combination of multiple interface connection methods.
[0061] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the head-mounted display device 100. While charging the battery 142, the charging management module 140 can also power the electronic device through the power management module 141.
[0062] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0063] The wireless communication function of the head display device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.
[0064] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the head-mounted display device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.
[0065] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use in the head-mounted display device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter, amplify, and perform other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be provided in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be provided in the same device as at least some of the modules of the processor 110.
[0066] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.
[0067] The wireless communication module 160 can provide wireless communication solutions for the head-mounted display device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), and other technologies. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, frequency modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.
[0068] In some embodiments, antenna 1 of the head-mounted display device 100 is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, so that the head-mounted display device 100 can communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).
[0069] The head-mounted display device 100 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.
[0070] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the head-mounted display device 100 may include one or N display screens 194, where N is a positive integer greater than 1.
[0071] The head-mounted display device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0072] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.
[0073] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or other format. In some embodiments, the head-mounted display device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0074] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the head-mounted display device 100 is selecting a frequency point, the digital signal processor is used to perform a Fourier transform on the frequency point energy.
[0075] Video codecs are used to compress or decompress digital video. The head-mounted display device 100 may support one or more video codecs. This allows the head-mounted display device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.
[0076] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in the head-mounted display device 100, such as image recognition, face recognition, speech recognition, and text comprehension.
[0077] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the head-mounted display device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored on the external memory card.
[0078] The internal memory 121 can be used to store computer-executable program code, which includes instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the head-mounted display device 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or a universal flash storage (UFS). The processor 110 executes various functional applications and data processing of the head-mounted display device 100 by executing instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.
[0079] The head-mounted display device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0080] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.
[0081] The speaker 170A, also called a "speaker," is used to convert audio electrical signals into sound signals. The head-mounted display device 100 can listen to music or make hands-free calls through the speaker 170A.
[0082] The receiver 170B, also called a "handset," is used to convert audio electrical signals into sound signals. When the head-mounted display device 100 receives a call or voice message, the user can place the receiver 170B close to the ear to listen to the voice.
[0083] Microphone 170C, also known as a "microphone" or "microphone," is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by placing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. The headset 100 can be equipped with at least one microphone 170C. In other embodiments, the headset 100 can be equipped with two microphones 170C, which can not only collect sound signals but also implement noise reduction. In other embodiments, the headset 100 can be equipped with three, four, or more microphones 170C to collect sound signals, reduce noise, identify sound sources, implement directional recording, and more.
[0084] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0085] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A can be located on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The head-mounted display device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 194, the head-mounted display device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The head-mounted display device 100 can also calculate the location of the touch based on the detection signal from the pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions.
[0086] The gyroscope sensor 180B can be used to determine the motion of the head-mounted display device 100. In some embodiments, the gyroscope sensor 180B can be used to determine the angular velocity of the head-mounted display device 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can also be used for image stabilization. The gyroscope sensor 180B can also be used for navigation and somatosensory gaming.
[0087] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the head-mounted display device 100 uses the air pressure value measured by the air pressure sensor 180C to calculate the altitude to assist in positioning and navigation.
[0088] The magnetic sensor 180D includes a Hall effect sensor. The head-mounted display device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the head-mounted display device 100 is a flip-type headset, the head-mounted display device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the leather cover or the flip cover, features such as automatic unlocking of the flip cover can be configured.
[0089] The accelerometer 180E can detect the magnitude of acceleration of the head-mounted display 100 in all directions (generally three axes). When the head-mounted display 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.
[0090] Distance sensor 180F is used to measure distance. The head-mounted display device 100 can measure distance using infrared or laser. In some embodiments, when photographing a scene, the head-mounted display device 100 can use distance sensor 180F to measure distance for rapid focusing.
[0091] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The head-mounted display device 100 emits infrared light outward through the light emitting diode. The head-mounted display device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the head-mounted display device 100. When insufficient reflected light is detected, the head-mounted display device 100 can determine that there is no object near the head-mounted display device 100. The head-mounted display device 100 can use the proximity light sensor 180G to detect when a user holds the head-mounted display device 100 close to their ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used to automatically unlock and lock the screen in leather case mode and pocket mode.
[0092] Ambient light sensor 180L is used to sense ambient light brightness. The head-mounted display device 100 can adaptively adjust the brightness of the display screen 194 based on the perceived ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether the head-mounted display device 100 is inside a packaging bag to prevent accidental touches.
[0093] The fingerprint sensor 180H is used to collect fingerprints. The head-mounted display device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, and so on. For example, the fingerprint sensor can be configured on the front of the head-mounted display device 100 (below the display screen 194), or on the back of the head-mounted display device 100 (below the rear camera). Alternatively, the fingerprint recognition function can be implemented by configuring the fingerprint sensor in the touch screen, that is, the fingerprint sensor can be integrated with the touch screen to implement the fingerprint recognition function of the head-mounted display device 100. In this case, the fingerprint sensor can be configured in the touch screen, can be a part of the touch screen, or can be configured in the touch screen in other ways. In addition, the fingerprint sensor can also be implemented as a full-panel fingerprint sensor, so that the touch screen can be regarded as a panel that can be used for fingerprint collection at any position. In some embodiments, the fingerprint sensor can process the collected fingerprint (e.g., whether the fingerprint is verified) and send it to the processor 110, which then performs corresponding processing based on the fingerprint processing results. In other embodiments, the fingerprint sensor may further send the collected fingerprint to the processor 110 so that the processor 110 can process the fingerprint (e.g., verify the fingerprint). The fingerprint sensor in the embodiments of the present application may use any type of sensing technology, including but not limited to optical, capacitive, piezoelectric, or ultrasonic sensing technology.
[0094] The temperature sensor 180J is used to detect temperature. In some embodiments, the head-mounted display device 100 uses the temperature detected by the temperature sensor 180J to implement temperature management strategies. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the head-mounted display device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature falls below another threshold, the head-mounted display device 100 heats the battery 142 to prevent the head-mounted display device 100 from abnormally shutting down due to low temperatures. In other embodiments, when the temperature falls below yet another threshold, the head-mounted display device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdowns due to low temperatures.
[0095] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied on or near the touch sensor. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the head-mounted display device 100, in a location different from that of the display screen 194.
[0096] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.
[0097] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The head-mounted display device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the head-mounted display device 100.
[0098] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0099] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.
[0100] The SIM card interface 195 is used to connect a SIM card. A SIM card can be connected to and disconnected from the head-mounted display device 100 by inserting or removing it from the SIM card interface 195. The head-mounted display device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. These cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The head-mounted display device 100 interacts with the network through the SIM card to implement functions such as call and data communications. In some embodiments, the head-mounted display device 100 uses an eSIM, or embedded SIM card. The eSIM card can be embedded in the head-mounted display device 100 and cannot be separated from it.
[0101] Although not shown in FIG1 b , the head display device 100 may further include a Bluetooth device, a positioning device, a flashlight, a micro-projection device, a near field communication (NFC) device, etc., which are not described in detail here.
[0102] The software system of the head-mounted display device 100 can adopt a layered architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the head-mounted display device 100.
[0103] FIG1c is a block diagram of the software structure of the head-mounted display device 100 according to an embodiment of the present invention.
[0104] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.
[0105] The application layer can include a series of application packages.
[0106] As shown in FIG1c , the application package may include applications such as phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.
[0107] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.
[0108] As shown in FIG1c , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.
[0109] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.
[0110] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0111] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.
[0112] The phone manager is used to provide communication functions for the head-mounted display device 100, such as the management of call status (including answering, hanging up, etc.).
[0113] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.
[0114] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.
[0115] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.
[0116] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.
[0117] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.
[0118] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0119] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.
[0120] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0121] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0122] A 2D graphics engine is a drawing engine for 2D drawings.
[0123] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, sensor driver, audio driver, and microphone driver.
[0124] The following embodiments can all be implemented in an electronic device having the above hardware structure (e.g., head-mounted display device 100). The following embodiments will take the head-mounted display device 100 as an example and describe in detail the information interaction method provided in the embodiments of the present application in conjunction with the accompanying drawings.
[0125] FIG2 exemplarily illustrates a flow diagram of an information interaction method provided in an embodiment of the present application. This method is applicable to the application scenario illustrated in FIG1a . The method flow can be implemented by a head-mounted display device alone, or can also be implemented through data interaction between a mobile terminal and the head-mounted display device. The information interaction method may include:
[0126] S201: The head-mounted display device enters a preset gesture image recording state, wherein the preset gesture in the preset gesture image can be any user-defined gesture.
[0127] In the embodiment of the present application, the head-mounted display device can enter the preset gesture image recording state in the following ways:
[0128] In some implementations, the head-mounted display device may receive a first voice message and determine, based on the first voice message, whether to trigger the head-mounted display device to enter a preset gesture image entry state.
[0129] In some embodiments, the head-mounted display device can directly receive the user's voice message as the first voice message. For example, if the user says "custom gesture entry" within the distance that the head-mounted display device can receive voice messages, the head-mounted display device receives the voice message and receives "custom gesture entry" as the first voice message.
[0130] In some embodiments, the head-mounted display device can receive a first voice message sent from a mobile terminal. For example, within the range of the mobile terminal that can receive voice messages, a user says "custom gesture entry." After receiving the voice message, the mobile terminal sends "custom gesture entry" to the head-mounted display device, which then receives "custom gesture entry" as the first voice message. In the embodiments of the present application, the mobile terminal can be a mobile phone, etc., and is not limited here.
[0131] After the head-mounted display device obtains the first voice message based on any of the above embodiments, it can perform voice recognition on the first voice message to determine whether the first preset word is contained in the first voice message. If the first preset word is contained, the head-mounted display device is triggered to enter the preset gesture image input state; if the first preset word is not contained, the head-mounted display device does not perform any processing.
[0132] The first preset word may include a first gesture type or a second gesture type. The first gesture type may be a gesture type corresponding to any specific function, for example, a gallery opening gesture, a call answering gesture, a gallery gesture, an incoming call gesture, etc. The second gesture type may be a gesture type corresponding to a non-specific function, for example, a custom gesture, etc.
[0133] For example, if the first voice message is "Gallery Gesture Entry", the head-mounted display device performs voice recognition on "Gallery Gesture Entry" and determines that the "Gallery Gesture Entry" contains the first preset word "Gallery Gesture", then the head-mounted display device can display the first user interface as shown in Figure 3. The first user interface includes four virtual buttons: Gallery Gesture, Open Gallery Gesture, Close Gallery Gesture, and Custom Operation Gesture. Among them, the "Gallery Gesture" virtual button is used to identify the gesture type, the "Open Gallery Gesture" virtual button is used to obtain the user's Gallery Open gesture, the "Close Gallery Gesture" virtual button is used to obtain the user's Gallery Close gesture, and the "Custom Operation Gesture" virtual button is used to obtain a gesture corresponding to any user-defined response function, for example, a user-defined Gallery Clear gesture.
[0134] If the first voice message is "Custom Gesture Entry," the head-mounted display device performs voice recognition on "Custom Gesture Entry," determines that "Custom Gesture Entry" contains the first preset word "Custom Gesture," and then displays the second user interface shown in FIG4 . The second user interface includes four virtual buttons: Custom Gesture, Gallery Gesture, Incoming Call Gesture, and Other Application Gestures. The "Custom Gesture" virtual button is used to identify the gesture type, the "Gallery Gesture" virtual button is used to retrieve gestures related to the gallery, the "Incoming Call Gesture" virtual button is used to retrieve gestures related to incoming calls, and the "Other Application Gesture" virtual button is used to retrieve gestures related to other applications, which can be any application installed on the head-mounted display device.
[0135] It should be understood that the first preset word in the head-mounted display device can be determined by the device manufacturer or by the user. The first preset word in the head-mounted display device can be fixed or continuously updated.
[0136] In some implementations, the head-mounted display device may receive a first keyboard message, where the first keyboard message is used to trigger the head-mounted display device to enter a preset gesture image entry state.
[0137] In some embodiments, the head-mounted display device may directly receive a first keyboard message from the user. For example, the user clicks a first key on a virtual keyboard of the head-mounted display device, and the head-mounted display device generates a first keyboard message in response to the click of the first key.
[0138] In some embodiments, the head-mounted display device may receive a first keyboard message from the mobile terminal. For example, a user clicks a first key on a keyboard of the mobile terminal. In response to the click of the first key, the mobile terminal generates a first keyboard message and sends the first keyboard message to the head-mounted display device.
[0139] It should be understood that the first button can be a virtual button on the virtual keyboard of the head display device, or a physical button on the physical keyboard of the mobile terminal.
[0140] In the embodiments of the present application, the first button is a virtual button on a virtual keyboard of a head-mounted display device. The first button can be a virtual button corresponding to a first gesture type, or a virtual button corresponding to a second gesture type. The first gesture type can be any specific gesture type, for example, a gallery opening gesture, an incoming call answering gesture, a gallery gesture, an incoming call gesture, etc. The second gesture type can be a non-specific gesture type, for example, a custom gesture, etc.
[0141] For example, if the first button is a “Gallery Gesture” virtual button, after the user clicks the “Gallery Gesture” virtual button, the head-mounted display device obtains the first keyboard message, and the head-mounted display device displays the first user interface as shown in FIG3 .
[0142] If the first button is a “custom gesture” virtual button, after the user clicks the “custom gesture” virtual button, the head display device obtains the first keyboard message, and the head display device displays the first user interface as shown in FIG4 .
[0143] It should be understood that before the mobile terminal sends the first voice message or the first keyboard message to the head-mounted display device, the mobile terminal may establish a connection with the head-mounted display device. The connection may be a wireless connection or a wired connection. If the mobile terminal and the head-mounted display device are wirelessly connected, it can be understood that the mobile terminal and the head-mounted display device are wirelessly connected via a wireless communication module in the mobile terminal and a wireless communication module in the head-mounted display device. For example, the mobile terminal and the head-mounted display device may be wirelessly connected via Wi-Fi or Bluetooth.
[0144] In step S202, the head-mounted display device obtains a preset gesture image, a preset response function, and a preset user name, and generates a first data set based on the preset gesture image and the preset user name, and generates a second data set based on the preset gesture image and the preset response function.
[0145] In an embodiment of the present application, the preset gesture image is determined based on the gesture made by the user corresponding to the preset user name. The preset gesture image corresponds to the preset response function, and the preset response function is the response made by the head-mounted display device to the preset gesture image.
[0146] In the case where the head-mounted display device displays the first user interface, the first user interface shown in Figure 5 is used as an example for explanation. As shown in Figure 5, if the user clicks the "Open Gallery Gesture" virtual button, the head-mounted display device activates the camera on the head-mounted display device and captures the user's gesture through the camera. The preset response function corresponding to the "Open Gallery Gesture" virtual button is to open the gallery, and the image containing the user's gesture captured by the camera is used as the preset gesture image. If the image containing the user's gesture captured by the camera is Figure 6, Figure 6 is used as the preset gesture image corresponding to the preset response function "Open Gallery", and Figure 6, with the "Open Gallery" function label, is added to the second dataset.
[0147] For other virtual buttons of the first user interface, other preset gesture images can also be obtained. The method of obtaining the preset gesture image is similar to the method of obtaining the preset gesture image for the above-mentioned "Open Gallery Gesture" virtual button, and is not limited here.
[0148] It should be understood that in order to ensure the accuracy of subsequent recognition, the preset gesture image corresponding to the preset response function obtained by the head-mounted display device can be a single preset gesture image or multiple preset gesture images of the same user, wherein the gestures in the multiple preset gesture images are relatively similar, but there are some differences.
[0149] For example, the headset captures two preset gesture images. The preset gesture images corresponding to the Gallery function are Figures 7 and 8. Figure 7 includes three joints: A, B, and C. Figure 8 includes three joints: E, F, and G. In Figure 7, the line segment between A and B is AB, the line segment between B and C is BC, and the angle between AB and BC is Angle 1. In Figure 8, the line segment between E and F is EF, the line segment between F and G is FG, and the angle between EF and FG is Angle 2. As can be seen, Angle 1 and Angle 2 are not equal. Therefore, for the Gallery function, the headset captures two preset gesture images. The gestures in these two preset gesture images are similar, but there are some differences.
[0150] When the head-mounted display device displays the second user interface, the second user interface shown in Figure 4 is used as an example for explanation. If the user clicks the "Gallery Gesture" virtual button, the head-mounted display device displays the first user interface shown in Figure 3. How to operate the first user interface shown in Figure 3 has been described in detail above and will not be repeated here.
[0151] In an embodiment of the present application, the head-mounted display device further needs to obtain a preset user name. After obtaining the preset user name, it determines whether the preset user name already exists. If the preset user name does not exist, a corresponding application space is allocated for the preset user name. If the preset user name already exists, the application space corresponding to the preset user name also exists, and there is no need to allocate an application space for the preset user name. The application space allocated for the preset user name includes different application software installed on the head-mounted display device. The user can access the application space corresponding to the preset user name and use the application software in the application space based on the preset user name. Moreover, the application space is only accessible to the user corresponding to the preset user name.
[0152] After obtaining the preset user name, the head display device adds the preset gesture image with the preset user name label to the first data set.
[0153] The head-mounted display device can obtain the preset user name through the following implementation.
[0154] In some embodiments, before executing S202, the head-mounted display device receives a second voice message, wherein the second voice message includes a preset user name. The head-mounted display device performs voice recognition on the second voice message to obtain the preset user name.
[0155] Alternatively, the first voice message received by the head display device includes a preset user name, and the head display device performs voice recognition on the first voice message to obtain the preset user name.
[0156] In some embodiments, before executing S202, the head-mounted display device receives a second keyboard message, and the head-mounted display device obtains a preset user name based on the second keyboard message. For example, a user enters a user name via a virtual keyboard of the head-mounted display device as the second keyboard message, and the head-mounted display device obtains the user name as the preset user name based on the second keyboard message.
[0157] Optionally, when the head-mounted display device obtains the preset gesture image, the preset response function, and the preset user name, the head-mounted display device may also collect the iris of the user corresponding to the preset user name, and generate a preset iris image based on the collected iris.
[0158] It should be understood that after the user puts on the head-mounted display device, an iris collector is installed on the inside of the head-mounted display device, which can directly collect the user's iris.
[0159] After acquiring the preset gesture image, the preset response function, the preset user name, and the preset iris image, the head-mounted display device generates a third data set based on the preset gesture image, the preset iris image, and the preset user name. Specifically, the head-mounted display device adds the preset gesture image tagged with the preset user name and the preset iris image tagged with the preset user name as a whole to the third data set.
[0160] S203: The head-mounted display device stores the first data set and the second data set.
[0161] The head-mounted display device can store the first data set and the second data set in the head-mounted display device and can also store them in a cloud server.
[0162] Optionally, after storing the first data set and the second data set, the head-mounted display device may perform the following two processing embodiments:
[0163] In some embodiments, the head-mounted display device trains a first identity re-identification model based on the first dataset and updates a gesture classification model based on the second dataset. Subsequently, the head-mounted display device can recognize the first gesture image of the first user based on the trained first identity re-identification model and the gesture classification model.
[0164] It should be understood that when the head-mounted display device leaves the factory, the head-mounted display device may store an initialized first identity re-identification model and gesture classification model. Alternatively, the head-mounted display device may not store the first identity re-identification model and gesture classification model, but instead download the initialized first identity re-identification model and gesture classification model from a cloud server when in use. The parameters in the initialized first identity re-identification model and gesture classification model may be random initial values or obtained through training using other datasets.
[0165] Taking a head-mounted display device (HMD) as an example, the HMD device stores an initialized first identity re-identification model. The HMD device obtains a first dataset and inputs the first dataset into the first identity re-identification model to train the first identity re-identification model. Preset gesture images in the first dataset serve as first training data, and preset usernames in the first dataset serve as first true labels. Specifically, the first identity re-identification model includes three modules: a first feature extraction module, a first classification module, and a first loss function module. During the training process of the first identity re-identification model, the first feature extraction module receives the preset gesture image and performs feature extraction on the preset gesture image to obtain first preset features. The first preset features are then output to the first classification module. The first classification module receives the first preset features and, based on the first preset features, outputs a first predicted label for the preset gesture image. The first loss function module receives the first predicted label and the first true label, calculates the difference between the first predicted label and the first true label to obtain a first loss value, and updates the parameters of the first feature extraction module, the first classification module, and the first loss function module in the first identity re-identification model based on the first loss value, thereby training the first identity re-identification model. It should be understood that the first preset feature may include at least one of the following: gesture joints and hand texture information. The gesture joints may include the 21 key points shown in FIG9 , and may also include other key points, without limitation. The hand texture information may include at least one of palm prints, skin texture, and other information, without limitation.
[0166] After the first identity re-identification model is trained, if the first gesture image of the first user is input into the first identity re-identification model, the first identity re-identification model can output a user name predicted for the first gesture image and use the user name as the first user name.
[0167] Taking the example of a head-mounted display device storing an initialized gesture classification model, the head-mounted display device obtains a second dataset and inputs the second dataset into the gesture classification model to train the gesture classification model. Preset gesture images in the second dataset serve as second training data, and preset response functions in the second dataset serve as second true labels. Specifically, the gesture classification model includes three modules: a second feature extraction module, a second classification module, and a second loss function module. During the training process of the gesture classification model, the second feature extraction module receives the preset gesture image and performs feature extraction on the preset gesture image to obtain second preset features. The second preset features are then output to the second classification module. The second classification model receives the second preset features and, based on the second preset features, outputs a second predicted label for the preset gesture image. The second loss function module receives the second predicted label and the second true label, calculates the difference between the second predicted label and the second true label, obtains a second loss value, and updates the parameters of the second feature extraction module, the second classification module, and the second loss function module in the gesture classification model based on the second loss value, thereby training the gesture classification model. It should be understood that the second preset features may include at least one of the following: gesture joints or hand texture information. The hand gesture joints may include the 21 key points shown in FIG9 , and may also include other key points, which are not limited here. The hand texture information may include at least one of palm prints, skin texture, etc., which are not limited here.
[0168] After the gesture classification model is trained, if a first gesture image of a first user is input into the gesture classification model, the gesture classification model may output a response function predicted for the first gesture image and use the response function as the first response function.
[0169] In an embodiment of the present application, different gesture classification models may be trained for different preset user names; or the same gesture classification model may be trained.
[0170] Optionally, the head-mounted display device may store the third dataset either on the head-mounted display device or on a cloud server. The head-mounted display device may also train a second identity re-identification model based on the third dataset. Subsequently, the head-mounted display device may recognize the first gesture image of the first user based on the trained second identity re-identification model and the gesture classification model.
[0171] It should be understood that the HMD may store an initialized second identity re-identification model when it leaves the factory. Alternatively, the HMD may not store the second identity re-identification model, but instead download the initialized second identity re-identification model from a cloud server when in use. The parameters in the initialized second identity re-identification model may be random initial values or obtained through training using other datasets.
[0172] Taking the example of a second identity re-identification model initialized and stored in a head-mounted display device, the head-mounted display device obtains a third dataset and inputs the third dataset into the second identity re-identification model for training. Preset gesture images and iris images in the third dataset serve as third training data, and the preset usernames in the third dataset serve as third ground-truth labels. Specifically, the second identity re-identification model includes four modules: a third feature extraction module, a fourth feature extraction module, a third classification module, and a third loss function module. During the training of the second identity re-identification model, the third feature extraction module receives a preset gesture image, performs feature extraction on the preset gesture image, obtains a third preset feature, and then outputs the third preset feature to the third classification module. The fourth feature extraction module receives a preset iris image, performs feature extraction on the preset iris image, obtains a fourth preset feature, and then outputs the fourth preset feature to the third classification module. The third classification module receives the third and fourth preset features and, based on the third and fourth preset features, outputs a third predicted label for the preset gesture image. The third loss function module receives the third predicted label and the third true label, calculates the difference between the third predicted label and the third true label, obtains a third loss value, and updates the parameters in the third feature extraction module, the fourth feature extraction module, the third classification module, and the third loss function module in the second identity re-identification model based on the third loss value, thereby realizing the training of the second identity re-identification model. It should be understood that the third preset feature may include at least one of the following: gesture joints, hand texture information. Among them, the gesture joints may include the 21 key points shown in Figure 9, and the gesture joints may also include other key points, which are not limited here. The hand texture information may be at least one information such as palm prints, skin texture, etc., which are not limited here.
[0173] After the second identity re-identification model is trained, if the first gesture image and the first iris image of the first user are input into the second identity re-identification model, the second identity re-identification model can output the user name predicted for the first gesture image and the first iris image as the first user name.
[0174] In some embodiments, the head-mounted display device uses a machine learning method to extract features from the preset gesture image to obtain a fifth preset feature. Subsequently, the head-mounted display device can recognize the first gesture image of the first user based on the fifth preset feature.
[0175] In the embodiment of the present application, the fifth preset feature may include at least one of the following: gesture joints and hand texture information. The gesture joints may include the 21 key points shown in FIG9 , and may also include other key points, without limitation. The hand texture information may include at least one of palm prints, skin texture, and the like, without limitation.
[0176] After obtaining the fifth preset feature, the head display device can associate the preset gesture image with the fifth preset feature to generate and store a fourth association relationship.
[0177] Optionally, the head-mounted display device may further employ a machine learning method to extract features from the preset iris image to obtain a sixth preset feature. Subsequently, the head-mounted display device may recognize the first gesture image of the first user based on the sixth preset feature, or based on both the fifth and sixth preset features.
[0178] After obtaining the sixth preset feature, the head display device can associate the preset iris image with the sixth preset feature to generate and store a fifth association relationship.
[0179] In addition, the head-mounted display device may also associate the preset gesture images in the first data set with the preset user name to generate and store a first association relationship. The head-mounted display device may also associate the preset gesture images in the second data set with the preset response function to generate and store a second association relationship. The head-mounted display device may also associate the preset gesture images, preset iris images, and preset user name in the third data set to generate and store a third association relationship.
[0180] It should be understood that the machine learning methods involved in the embodiments of the present application are not specifically limited.
[0181] When a user is using a head-mounted display (HMD) and receives an incoming call on their mobile terminal, the current standard practice is for the user to remove the HMD and then answer the call on their mobile terminal. If the user wants to make a call using their mobile terminal, they must remove the HMD and then dial the number on their mobile terminal. To facilitate convenient phone calls, the HMD provided in this embodiment of the present application has a mobile communication function. To implement this function, this embodiment of the present application provides the following two possible implementation methods.
[0182] In some embodiments, the head-mounted display device is connected to a mobile terminal, and the head-mounted display device can directly and remotely use the communication function of the mobile terminal.
[0183] It should be understood that the head-mounted display device and the mobile terminal can be connected via a wired connection or a wireless connection. If the mobile terminal and the head-mounted display device are connected wirelessly, it can be understood that the wireless connection between the mobile terminal and the head-mounted display device is achieved through a wireless communication module in the mobile terminal and a wireless communication module in the head-mounted display device. For example, the mobile terminal and the head-mounted display device can be wirelessly connected via WiFi or Bluetooth.
[0184] After the headset is connected to the mobile terminal, the headset can utilize the mobile terminal's communication functions. For example, if the mobile terminal receives an incoming call, both the mobile terminal and the headset can display the call interface shown in Figure 10. Without removing the headset, the user can tap the "Accept" virtual button on the call interface displayed on the headset to initiate the call, or tap the "Reject" virtual button to reject the call.
[0185] In an embodiment of the present application, a user may click a virtual button on a call interface displayed on a head-mounted display device through the following possible implementations: the user may operate through the handle 101 in FIG. 1a , may operate directly through a finger, or may operate through a preset gesture set by the user.
[0186] Taking the example of a user operating through preset gestures, the call answering gesture corresponding to the "Answer Call" function is shown in Figure 11, and the call rejecting gesture corresponding to the "Reject Call" function is shown in Figure 12. When the head-mounted display device displays the call interface shown in Figure 10, if the user performs the gesture shown in Figure 11 within the visible area of the head-mounted display device, it is equivalent to clicking the "Accept" virtual button on the call interface displayed by the head-mounted display device, and the mobile terminal can communicate; if the user performs the gesture shown in Figure 12 within the visible area of the head-mounted display device, it is equivalent to clicking the "Reject" virtual button on the call interface displayed by the head-mounted display device, and the mobile terminal rejects the communication. Regardless of whether the mobile terminal communicates, the above call records can be saved in the head-mounted display device or not.
[0187] Optionally, the address book, call logs, etc. in the mobile terminal can be uploaded to the head-mounted display device for storage. The user can also manually update the address book using the head-mounted display device's virtual keyboard. For example, the user can enter a contact name and contact phone number in the head-mounted display device to update the address book.
[0188] In some embodiments, the head-mounted display device includes a mobile communication module 150, shown in FIG1b , which enables the head-mounted display device to have communication capabilities. A SIM card can be inserted into the SIM card interface 195 of the head-mounted display device, enabling the head-mounted display device to dial a number. The number associated with the SIM card in the head-mounted display device and the number associated with the SIM card in the mobile terminal can be the same or different.
[0189] For example, if the head-mounted display device receives an incoming call, the head-mounted display device can display a call interface as shown in Figure 10. The user can click the "Accept" virtual button in the call interface displayed on the head-mounted display device to communicate; or click the "Reject" virtual button to reject the communication.
[0190] Optionally, the head-mounted display device can be connected to a mobile terminal, where the connection between the head-mounted display device and the mobile terminal can be wired or wireless. After the head-mounted display device and the mobile terminal are connected, the address book, call logs, etc. in the mobile terminal can be uploaded to the head-mounted display device for storage. The user can also manually update the address book using the head-mounted display device's virtual keyboard. For example, the user can enter a contact name and contact phone number in the head-mounted display device to update the address book.
[0191] Through the above method, when a user is using the head-mounted display device and receives an incoming call, the user can answer the call directly using the head-mounted display device without removing the head-mounted display device. Similarly, if the user wants to make a call, the user can also make the call directly using the head-mounted display device without removing the head-mounted display device. Therefore, the head-mounted display device can provide convenient mobile communication functions.
[0192] In order to protect the privacy of users using a head-mounted display device, an embodiment of the present application provides an information interaction method, that is, when a first user uses a head-mounted display device, the head-mounted display device displays a first application space corresponding to the first user, so that the first user can access the first application space and use the application software in the first application space. At the same time, in order to improve the convenience of using the head-mounted display device, the first user can conveniently use the application software in the first application space through customized gestures, wherein the first user can be a user corresponding to any preset user name. Therefore, when the first user uses the head-mounted display device, the head-mounted display device can perform the following steps as shown in Figure 13:
[0193] S1301: The head-mounted display device obtains a first gesture image of a first user, and determines a first application space and a first response function corresponding to the first gesture image.
[0194] The first gesture image may include one or more gesture images, without limitation. The first application space is the application space allocated by the head-mounted display device for the first user, and the first response function is the response of the head-mounted display device to the first gesture image.
[0195] The head-mounted display device can determine the first application space and the first response function through the following possible implementation methods.
[0196] In an embodiment of the present application, the head-mounted display device can determine the first application space based on the first identity re-identification model, and determine the first response function based on the gesture classification model.
[0197] The head-mounted display device determines the first application space based on the first identity re-identification model, which can include the following steps: the head-mounted display device first determines the first user name corresponding to the first gesture image based on the first identity re-identification model, and uses the application space corresponding to the first user name as the first application space, wherein the first identity re-identification model can be obtained through training of the first data set.
[0198] For example, if the first identity re-identification model is trained based on the first dataset, then after the head-mounted display device obtains the first gesture image of the first user, it inputs the first gesture image into the first identity re-identification model. The first identity re-identification model includes three modules: a first feature extraction module, a first classification module, and a first loss function module. The first loss function module is only used when training the first identity re-identification model. The first feature extraction module receives the first gesture image, performs feature extraction on the first gesture image, obtains a first feature, and then outputs the first feature to the first classification module. The first classification module receives the first feature and, based on the first feature, outputs a fourth predicted label for the first gesture image. The fourth predicted label is the first user name.
[0199] It should be understood that the first feature may include at least one of the following: a gesture joint point, hand texture information. The first feature corresponds to the first preset feature. For example, if the first preset feature is a gesture joint point, the first feature is also a gesture joint point.
[0200] Through the above method, the head display device inputs the first gesture image into the first identity re-identification model, and can determine the first user name based only on the first gesture image, which can effectively improve the recognition efficiency.
[0201] The head-mounted display device can also determine the first application space based on the second identity re-identification model, which can include the following steps: the head-mounted display device determines the first user name corresponding to the first gesture image based on the second identity re-identification model, and uses the application space corresponding to the first user name as the first application space, wherein the second identity re-identification model can be obtained through training with a third data set.
[0202] For example, if the second identity re-identification model is trained based on a third dataset, after acquiring the first gesture image of the first user, the head-mounted display device may also acquire the first user's iris image as the first iris image. The head-mounted display device inputs the first gesture image and the first iris image into the second identity re-identification model. The second identity re-identification model includes four modules: a third feature extraction module, a fourth feature extraction module, a third classification module, and a third loss function module. The third loss function module is only used when training the second identity re-identification model. The third feature extraction module receives the first gesture image and performs feature extraction on the first gesture image to obtain a third feature, which is then output to the third classification module. The fourth feature extraction module receives the first iris image and performs feature extraction on the first iris image to obtain a fourth feature, which is then output to the third classification module. The third classification module receives the third and fourth features and, based on the third and fourth features, outputs a fifth predetermined label for the first gesture image. This fifth predicted label is the first user's name.
[0203] It should be understood that the third feature may include at least one of the following: gesture joints, hand texture information. The third feature corresponds to the third preset feature. For example, if the third preset feature is a gesture joint, then the third feature is also a gesture joint.
[0204] Through the above method, the head display device inputs the first gesture image and the first iris image into the second identity re-identification model, and determines the first user name based on the first gesture image and the first iris image, which can effectively improve the recognition accuracy.
[0205] The head-mounted display device determines the first response function based on the gesture classification model, which may include the following steps: the head-mounted display device determines the first response function corresponding to the first gesture image based on the gesture classification model, wherein the gesture classification model can be obtained through training of the second data set.
[0206] For example, if the gesture classification model is trained based on the second dataset, after the head-mounted display device obtains a first gesture image of the first user, it inputs the first gesture image into the gesture classification model. The gesture classification model includes three modules: a second feature extraction module, a second classification module, and a second loss function module. The second loss function module is only used during training of the gesture classification model. The second feature extraction module receives the first gesture image, performs feature extraction on the first gesture image, obtains a second feature, and then outputs the second feature to the second classification module. The second classification module receives the second feature and, based on the second feature, outputs a sixth predicted label for the first gesture image. This sixth predicted label is the first response function.
[0207] It should be understood that the second feature may include at least one of the following: a gesture joint point, hand texture information. The second feature corresponds to the second preset feature. For example, if the second preset feature is a gesture joint point, the second feature is also a gesture joint point.
[0208] Through the above method, the head display device inputs the first gesture image into the gesture classification model, and determines the first response function according to the first gesture image, thereby providing a convenient method for determining the first response function.
[0209] In the embodiment of the present application, as the user becomes more and more proficient in using the head-mounted display device, the gestures presented when using the head-mounted display device will become relatively less and less standard; or, as the user ages, the user's hand texture information will gradually change. In order to effectively ensure the accuracy of the first identity re-identification model, the gesture classification model and the second identity re-identification model, the above models need to be updated regularly.
[0210] After determining the first user name and the first response function corresponding to the first gesture image, the head-mounted display device can update the first identity re-identification model, the gesture classification model and the second identity re-identification model according to the first gesture image, the first iris image, the first user name and the first response function.
[0211] In some embodiments, the head-mounted display device can directly add the first gesture image with the first username label to the first data set, and use the updated first data to train the first identity re-identification model.
[0212] In some embodiments, the head-mounted display device may further determine at least one preset gesture image tagged with a first user name from the first data set, and use the determined preset gesture image tagged with the first user name as a first candidate gesture image. A first similarity between the at least one first candidate gesture image and the first gesture image is then determined, and the following possible implementations may be performed based on the first similarity:
[0213] In some embodiments, if the first similarity is less than a first threshold, the head-mounted display device adds the first gesture image labeled with the first username to the first dataset and trains the first identity re-identification model using the updated first dataset. If the first similarity is greater than or equal to the first threshold, the head-mounted display device does not perform any further processing.
[0214] The updated first data set obtained by the above implementation method does not include gesture images with a high similarity, so when the updated first data set is used to train the first identity re-identification model, it is beneficial to improve the accuracy of the first identity re-identification model.
[0215] In some embodiments, when the first similarity is less than a first threshold and greater than a second threshold, the head-mounted display device adds the first gesture image labeled with the first username to the first dataset and uses the updated first dataset to train the first identity re-identification model. When the first similarity is greater than or equal to the first threshold, or when the first similarity is less than or equal to the second threshold, the head-mounted display device does not perform any further processing. The second threshold is less than the first threshold.
[0216] In the above method, if the first similarity is greater than or equal to the first threshold, it indicates that the first gesture image and at least one first candidate gesture image are highly similar, and the first gesture image tagged with the first username may not be used to update the first dataset. If the first gesture image tagged with the first username is added to the first dataset, due to the high similarity between the first gesture image and at least one first candidate gesture image, using a large number of similar gesture images to train the first identity re-identification model may cause overfitting of the first identity re-identification model, thereby affecting the accuracy of the first identity re-identification model.
[0217] If the first similarity is less than or equal to the second threshold, the similarity between the first gesture image and the at least one first candidate gesture image is low, and the first gesture image tagged with the first username may not be used to update the first dataset. If the first gesture image tagged with the first username is added to the first dataset, due to the low similarity between the first gesture image and the at least one first candidate gesture image, the first gesture image will be equivalent to noise data relative to the other first candidate gesture images, potentially affecting the subsequent training of the first identity re-identification model and, in turn, the accuracy of the first identity re-identification model.
[0218] The updated first data set obtained through the above implementation method does not include gesture images with a high similarity or noise data. Therefore, when using the updated first data set to train the first identity re-identification model, it is more conducive to improving the accuracy of the first identity re-identification model.
[0219] In some embodiments, when the first similarity is less than a first threshold, or when the first similarity is less than the first threshold and greater than a second threshold, before the head-mounted display device adds the first gesture image tagged with the first username to the first dataset, the head-mounted display device may determine whether the number of first candidate gesture images is less than a first threshold. If the number of first candidate gesture images is less than the first threshold, the head-mounted display device adds the first gesture image tagged with the first username to the first dataset. If the number of first candidate gesture images is greater than or equal to the first threshold, the head-mounted display device deletes a preset gesture image tagged with the first username from the first dataset and adds the first gesture image tagged with the first username to the first dataset. The deleted preset gesture image tagged with the first username may be randomly selected or may have the least similarity to the first gesture image. Through the above method, the storage space occupied by the first dataset can be limited. Moreover, deleting the preset gesture image with the least similarity to the first gesture image from the first dataset is equivalent to removing noise data from the first dataset, which is beneficial to improving the accuracy of the first identity re-identification model.
[0220] In an embodiment of the present application, the first similarity can be determined in the following possible ways: the head-mounted display device separately determines the similarity between each first candidate gesture image and the first gesture image, and takes the average of the determined similarities as the first similarity; or, the head-mounted display device separately determines the similarity between each first candidate gesture image and the first gesture image, and takes the determined minimum similarity as the first similarity.
[0221] It should be understood that when calculating the similarity between the first candidate gesture image and the first gesture image, the similarity can be calculated based on the features of the first candidate gesture image and the features of the first gesture image. The features of the first candidate gesture image and the features of the first gesture image can include at least one of the following: gesture joints and hand texture information. The features of the first candidate gesture image correspond to the features of the first gesture image. For example, if the features of the first candidate gesture image are gesture joints, then the features of the first gesture image are also gesture joints.
[0222] Through the above method, when the gestures made by the user are not standard, or when the texture information of the user's hand changes as the user ages, the first identity re-identification model can still perform accurate recognition.
[0223] In some embodiments, the head-mounted display device can directly add the first gesture image with the first response function label to the second dataset, and use the updated second dataset to train the gesture classification model.
[0224] In some embodiments, the head-mounted display device may further determine at least one preset gesture image with a first response function label from the second data set, and use the determined preset gesture image with the first response function label as a second candidate gesture image. A second similarity between the at least one second candidate gesture image and the first gesture image is then determined, and the following possible implementations may be performed based on the second similarity:
[0225] In some embodiments, if the second similarity is less than a third threshold, the head-mounted display device adds the first gesture image with the first response function label to the second dataset and uses the updated second dataset to train the gesture classification model. If the second similarity is greater than or equal to the third threshold, the head-mounted display device does not perform any further processing.
[0226] In some embodiments, when the second similarity is less than a third threshold and greater than a fourth threshold, the head-mounted display device adds the first gesture image with the first response function label to the second dataset and uses the updated second dataset to train the gesture classification model. When the second similarity is greater than or equal to the third threshold, or when the second similarity is less than or equal to the fourth threshold, the head-mounted display device performs no further processing. The fourth threshold is less than the third threshold.
[0227] The above method of obtaining the updated second data set is beneficial to improving the accuracy of the gesture classification model. The specific analysis process can refer to the method of obtaining the updated first data set, which is not described here in detail.
[0228] In some embodiments, when the second similarity is less than a third threshold, or when the second similarity is less than the third threshold and greater than a fourth threshold, before the head-mounted display device adds the first gesture image with the first response function label to the second dataset, the head-mounted display device may determine whether the number of second candidate gesture images is less than a second threshold. If the number of second candidate gesture images is less than the second threshold, the head-mounted display device adds the first gesture image with the first response function label to the second dataset. If the number of second candidate gesture images is greater than or equal to the second threshold, the head-mounted display device deletes a preset gesture image with the first response function label from the second dataset and adds the first gesture image with the first response function label to the second dataset. The deleted preset gesture image with the first response function label may be randomly selected or may have the least similarity to the first gesture image. Through the above method, the storage space occupied by the second dataset can be limited. Moreover, deleting the preset gesture image with the least similarity to the first gesture image from the second dataset is equivalent to removing noise data from the second dataset, which is beneficial for improving the accuracy of the gesture classification model.
[0229] In the embodiment of the present application, the method for determining the second similarity is similar to the method for determining the first similarity, and is not described in detail here.
[0230] Through the above method, the gesture classification model can still accurately identify the gestures made by the user even when the gestures made by the user are not standard, or when the texture information of the user's hand changes as the user ages.
[0231] In some embodiments, the head-mounted display device can directly add the first gesture image with the first username label and the first iris image with the first username label to the third dataset, and use the updated third dataset to train the second identity re-identification model.
[0232] In some embodiments, the head-mounted display device may further determine at least one preset gesture image tagged with the first user name from the third data set, and use the determined preset gesture image tagged with the first user name as a third candidate gesture image. A third similarity between the at least one third candidate gesture image and the first gesture image may then be determined. Based on the third similarity, the following possible implementations may be performed:
[0233] In some embodiments, if the third similarity is less than a fifth threshold, the head-mounted display device adds the first hand gesture image tagged with the first username and the first iris image tagged with the first username to a third dataset and uses the updated third dataset to train the second identity re-identification model. If the third similarity is greater than or equal to the fifth threshold, the head-mounted display device does not perform any further processing.
[0234] In some embodiments, when the third similarity is less than the fifth threshold and greater than the sixth threshold, the head-mounted display device adds the first gesture image tagged with the first username and the first iris image tagged with the first username to a third dataset and uses the updated third dataset to train the second identity re-identification model. When the third similarity is greater than or equal to the fifth threshold, or when the third similarity is less than or equal to the sixth threshold, the head-mounted display device does not perform any further processing. The sixth threshold is less than the fifth threshold.
[0235] The above method of obtaining the updated third data set is conducive to improving the accuracy of the second identity re-identification model. The specific analysis process can refer to the method of obtaining the updated first data set, which is not repeated here.
[0236] In some embodiments, when the third similarity is less than a fifth threshold, or when the third similarity is less than the fifth threshold and greater than a sixth threshold, before the head-mounted display device adds the first gesture image with the first username tag and the first iris image with the first username tag to the third data set, the head-mounted display device may determine whether the number of third candidate gesture images is less than a third number threshold. If the number of third candidate gesture images is less than the third number threshold, the head-mounted display device adds the first gesture image with the first username tag and the first iris image with the first username tag to the third data set; if the number of third candidate gesture images is greater than or equal to the third number threshold, the head-mounted display device deletes a preset gesture image with the first username tag and the corresponding preset iris image with the first username tag from the third data set, and adds the first gesture image with the first username tag and the first iris image with the first username tag to the third data set, wherein the deleted preset gesture image with the first username tag may be randomly selected or may have the least similarity to the first gesture image. Through the above method, the storage space occupied by the third data set can be limited. Moreover, if the preset gesture image with the least similarity to the first gesture image is deleted from the third data set, it is equivalent to eliminating the noise data in the third data set, which is beneficial to improving the accuracy of the second identity re-identification model.
[0237] In some embodiments, the head-mounted display device may further determine at least one preset iris image tagged with the first username from the third dataset, and use the determined preset iris image tagged with the first username as a fourth candidate gesture image. A fourth similarity between the at least one fourth candidate gesture image and the first gesture image is then determined, and the following possible implementations may be performed based on the fourth similarity:
[0238] In some embodiments, when the fourth similarity is less than a seventh threshold, the head-mounted display device adds the first hand gesture image tagged with the first username and the first iris image tagged with the first username to a third dataset and uses the updated third dataset to train the second identity re-identification model. When the fourth similarity is greater than or equal to the seventh threshold, the head-mounted display device does not perform any further processing.
[0239] In some embodiments, when the fourth similarity is less than the seventh threshold and greater than the eighth threshold, the head-mounted display device adds the first gesture image tagged with the first username and the first iris image tagged with the first username to the third dataset and uses the updated third dataset to train the second identity re-identification model. When the fourth similarity is greater than or equal to the seventh threshold, or when the fourth similarity is less than or equal to the eighth threshold, the head-mounted display device does not perform any further processing. The eighth threshold is less than the seventh threshold.
[0240] In some embodiments, the head-mounted display device may also make a judgment based on the third candidate gesture image and the fourth candidate gesture image simultaneously. Specifically, if the head-mounted display device determines that the third similarity is less than the fifth threshold and greater than the sixth threshold, and that the fourth similarity is less than the seventh threshold and greater than the eighth threshold, the head-mounted display device adds the first gesture image tagged with the first username and the first iris image tagged with the first username to the third dataset and uses the updated third dataset to update the second identity re-identification model. In other cases, the head-mounted display device does not perform any other processing.
[0241] In the embodiment of the present application, the method for determining the third similarity and the fourth similarity is similar to the method for determining the first similarity, and will not be described in detail here.
[0242] Through the above method, when the gestures made by the user are not standard, or when the texture information of the user's hand changes as the user ages, or when the user's iris changes, the second identity re-identification model can still perform accurate recognition.
[0243] In an embodiment of the present application, the head display device can determine the first application space and the first response function based on the fifth preset feature.
[0244] In some embodiments, the head-mounted display device uses a machine learning method to extract features from the first gesture image to obtain a fifth feature, and then calculates the similarity between the fifth feature and at least one fifth preset feature. For any fifth preset feature, if the similarity between the fifth preset feature and the fifth feature is greater than a ninth threshold, the fifth preset feature is used as a candidate feature. The candidate feature with the highest similarity is determined from the at least one candidate feature, and based on the fourth association relationship, the preset gesture image corresponding to the candidate feature with the highest similarity is determined. Based on the first association relationship or the third association relationship, the preset username corresponding to the preset gesture image is determined, and the preset username is used as the first username, wherein the application space corresponding to the first username is the first application space. Based on the second association relationship, the preset response function corresponding to the preset gesture image is determined, and the preset response function is used as the first response function.
[0245] It should be understood that the fifth feature may include at least one of the following: gesture joints, hand texture information. The fifth feature corresponds to the fifth preset feature. For example, if the fifth preset feature is a gesture joint, then the fifth feature is also a gesture joint.
[0246] In some embodiments, the head-mounted display device uses a machine learning method to extract features from the first iris image to obtain a sixth feature, and then calculates the similarity between the sixth feature and at least one sixth preset feature. For any sixth preset feature, if the similarity between the sixth preset feature and the sixth feature is greater than a tenth threshold, the sixth preset feature is used as a candidate feature. The candidate feature with the highest similarity is determined from the at least one candidate feature, and based on the fifth association relationship, the preset iris image corresponding to the candidate feature with the highest similarity is determined. Based on the third association relationship, the preset user name and preset gesture image corresponding to the preset iris image are determined, and the preset user name is used as the first user name, wherein the application space corresponding to the first user name is the first application space. Based on the second association relationship, the preset response function corresponding to the preset gesture image is determined, and the preset response function is used as the first response function.
[0247] It should be understood that the sixth feature may include at least one of the following: gesture joints, hand texture information. The sixth feature corresponds to the sixth preset feature. For example, if the sixth preset feature is a gesture joint, then the sixth feature is also a gesture joint.
[0248] In some embodiments, the head-mounted display device may also determine the first application space and the first response function based on the first gesture image and the first iris image. The specific method can be obtained by combining the above content and is not detailed here.
[0249] It should be understood that the machine learning method used by the head-mounted display device to obtain the fifth preset feature is the same as the machine learning method used by the head-mounted display device to obtain the fifth feature; the machine learning method used by the head-mounted display device to obtain the sixth preset feature is the same as the machine learning method used by the head-mounted display device to obtain the sixth feature.
[0250] In the above method, the head-mounted display device does not need to store the identity re-identification model (including the first identity re-identification model and / or the second identity re-identification model) and the gesture classification model, which saves the storage space of the head-mounted display device. In addition, the above method directly calculates the similarity based on the features, which can conveniently determine the first application space and the first response function, simplifying the complexity of the process.
[0251] In some embodiments, when a first user is using a head-mounted display device, if the head-mounted display device obtains both a first gesture image of the first user and a second gesture image of the second user, since the first user is using the head-mounted display device, the head-mounted display device can obtain the iris image of the first user, that is, the first iris image, and the head-mounted display device determines the corresponding first application space and first response function based on the first gesture image and the first iris image, rather than determining the corresponding second application space and second response function based on the second gesture image of the second user.
[0252] S1302: The head-mounted display device executes a first response function in the first application space.
[0253] The head-mounted display device starts a first application space corresponding to the first user name and executes a first response function in the first application space.
[0254] For example, the "Open Gallery" function corresponding to the gesture for opening the gallery is illustrated in FIG6 . If a first user performs the gesture shown in FIG14 within the visible area of the head-mounted display device (the gestures in FIG6 and FIG14 are similar), the head-mounted display device obtains a first gesture image including the gesture in FIG14 , determines the first user name corresponding to the first gesture image, and the "Open Gallery" function corresponding to the first gesture image, wherein the application space corresponding to the first user name is the first application space. The head-mounted display device executes the "Open Gallery" function in the first application space.
[0255] In the above method, the first user can access the application software in the first application space. At the same time, the first application space is only accessible to the first user, effectively protecting the privacy of the first user. Because different users correspond to different application spaces, different application spaces can be isolated, thereby protecting the privacy of different users.
[0256] In this embodiment of the present application, when a first user uses a new head-mounted display device, the first user can directly transfer information in the first application space of the original head-mounted display device to the new head-mounted display device. The information in the first application space may include pictures, address book, call logs, application software browsing history, etc. In addition, the information in the first application space may also include the associations related to the first user from the first to fifth associations, as well as the first identity re-identification model, the second identity re-identification model, and the gesture classification model.
[0257] It is understood that when the original head-mounted display device transfers the information in the first application space to the new head-mounted display device, the original head-mounted display device can be triggered based on the first gesture image, where the first gesture image includes the transfer gesture; it can also be triggered based on a virtual key on the virtual keyboard.
[0258] Optionally, if the information in the first application space of the original head-mounted display device is stored in the original head-mounted display device, when the original head-mounted display device transfers the information in the first application space to the new head-mounted display device, the original head-mounted display device may also be connected to the new head-mounted display device, wherein the original head-mounted display device and the new head-mounted display device may be connected by wired connection or wireless connection. If the original head-mounted display device and the new head-mounted display device are connected wirelessly, it can be understood that the wireless connection between the original head-mounted display device and the new head-mounted display device is achieved through the wireless communication module in the original head-mounted display device and the wireless communication module in the new head-mounted display device. For example, the original head-mounted display device and the new head-mounted display device may be wirelessly connected via WIFI or Bluetooth.
[0259] If the information in the first application space of the original head-mounted display device is stored in a cloud server, when the original head-mounted display device transfers the information in the first application space to the new head-mounted display device, the original head-mounted display device is connected to the cloud server, and the new head-mounted display device is connected to the cloud server. The above connection methods are all wireless. The original head-mounted display device may or may not be connected to the new head-mounted display device, and this is not limited here.
[0260] With this method, when a user uses a new head-mounted display device, they can quickly and easily transfer information from their own application space. This prevents information leakage because only the information in their own application space is transferred, while information from other users' application spaces is not.
[0261] In the embodiments provided above, the methods provided in the embodiments of this application are described from the perspective of a head-mounted display device as the execution subject. To implement the various functions of the methods provided in the embodiments of this application, the head-mounted display device may include hardware structures and / or software modules, and the aforementioned functions may be implemented in the form of hardware structures, software modules, or hardware structures and software modules. Whether a particular function is implemented in the form of hardware structures, software modules, or hardware structures and software modules depends on the specific application and design constraints of the technical solution.
[0262] Exemplarily, when hardware implementation is adopted, the hardware implementation of the head display device can refer to Figure 15 and its related description.
[0263] 15 , the head-mounted display device may include: one or more processors 1502; a memory 1503; one or more application programs (not shown); and one or more computer programs 1504. The aforementioned components may be connected via one or more communication buses 1501. The one or more computer programs 1504 are stored in the memory 1503 and configured to be executed by the one or more processors 1502. The one or more computer programs 1504 include instructions, which may be used to execute the method in any of the aforementioned embodiments.
[0264] An embodiment of the present application further provides a computer storage medium storing computer instructions. When the computer instructions are executed on a head-mounted display device, the head-mounted display device executes the above-mentioned related method steps to implement the information interaction method of the head-mounted display device in the above-mentioned embodiment.
[0265] An embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the information interaction method of the head-mounted display device in the above-mentioned embodiment.
[0266] Among them, the head display device, computer storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.
[0267] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0268] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0269] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0270] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0271] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0272] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. An information interaction method, applied to a head-mounted display device, characterized in that: The method comprises: Acquire a first gesture image of a first user, and determine a first application space and a first response function corresponding to the first gesture image, where the first application space is an application space allocated by a head-mounted display device to the first user, and the first response function is a response made by the head-mounted display device to the first gesture image; In the first application space, the first response function is executed.
2. The method according to claim 1, characterized in that The determining a first application space corresponding to the first gesture image includes: The first gesture image is input into a first identity re-identification model to obtain a first user name, and an application space corresponding to the first user name is used as the first application space, wherein the first identity re-identification model is obtained by training a first data set, and the first data set includes preset gesture images with preset user name labels.
3. The method according to claim 1, characterized in that The determining a first response function corresponding to the first gesture image includes: The first gesture image is input into a gesture classification model to obtain the first response function, wherein the gesture classification model is obtained by training with a second data set, and the second data set includes preset gesture images with preset response function labels.
4. The method according to claim 1, characterized in that After obtaining the first gesture image of the first user, the method further includes: Acquire a first iris image of the first user; The determining a first application space corresponding to the first gesture image includes: The first gesture image and the first iris image are input into a second identity re-identification model to obtain a first user name, and the application space corresponding to the first user name is used as the first application space. The second identity re-identification model is obtained by training a third data set, and the third data set includes a preset gesture image with a preset user name label, and a preset iris image with a preset user name label.
5. The method according to claim 2, characterized in that Also includes: The first gesture image with the first user name label is added to the first data set, and the first identity re-identification model is trained using the updated first data set.
6. The method according to claim 5, characterized in that The step of adding the first gesture image with the first user name label to the first data set includes: Determine at least one preset gesture image with the first user name label from the first data set; Determining a first similarity between the at least one preset gesture image with the first user name tag and the first gesture image; When the first similarity is smaller than a first threshold and larger than a second threshold, the first gesture image with the first user name label is added to the first data set.
7. The method according to claim 3, characterized in that Also includes: The first gesture image with the first response function label is added to the second data set, and the gesture classification model is trained using the updated second data set.
8. The method according to claim 7, characterized in that The step of adding the first gesture image with the first response function label to the second data set includes: Determining at least one preset gesture image with the first response function label from the second data set; determining a second similarity between the at least one preset gesture image with the first response function label and the first gesture image; When the second similarity is less than a third threshold and greater than a fourth threshold, the first gesture image with the first response function label is added to the second data set.
9. The method according to claim 4, characterized in that Also includes: The first gesture image with the first user name label and the first iris image with the first user name label are added to the third data set, and the second identity re-identification model is trained using the updated third data set.
10. The method according to claim 9, characterized in that The step of adding the first gesture image with the first user name label to the third data set includes: Determine at least one preset gesture image with the first user name label from the third data set; determining a third similarity between the at least one preset gesture image with the first user name tag and the first gesture image; When the third similarity is smaller than a fifth threshold and larger than a sixth threshold, the first gesture image with the first user name tag is added to the third data set.
11. A head mounted display device, characterized in that: including a processor and a memory; The processor is used to execute the computer program or instructions stored in the memory, so that the head display device implements the method described in any one of claims 1-10.
12. A computer-readable storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is executed on a computer, the computer-readable storage medium enables the computer-readable storage medium to implement the method according to any one of claims 1 to 10.
Citation Information
Patent Citations
User identification by gesture recognition
CN103733614A
Non-contact gesture command authentication method and user device
CN103995998A
Control method and device of multimedia equipment, electronic equipment and storage medium
CN113359978A
Human-computer interaction method and device, storage medium and electronic equipment
CN115291724A
Method and system for realizing XR gesture input
CN115407878A
