Information interaction method and head-mounted display device
The headset device uses identity re-identification and gesture classification models to identify the user's gesture images, determine its application space and response functions, and solves the problem of privacy protection in multiple people sharing headset devices, realizing effective protection of user privacy.
Patent Information
- Application Number
- CN202310548994.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-15
AI Technical Summary
When multiple family members share headset devices, how to effectively protect user privacy and prevent privacy leakage.
The user's gesture image is obtained through the headset device, and the identity recognition model and gesture classification model are used to determine the user's application space and response functions, ensuring that only authorized users can access specific application spaces and achieve privacy protection.
It effectively improves the protection of user privacy, prevents unauthorized access, and improves the security of user privacy.
Smart Images

Figure CN118963533A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of virtual reality technology, and in particular to an information interaction method and a head display device. Background Art
[0002] Virtual reality (VR) and augmented reality (AR) are emerging multimedia technologies in recent years. VR technology refers to presenting virtual information to users without them being able to see the real world, allowing users to completely detach from the real scene and place themselves in the virtual world. AR technology refers to superimposing virtual information on the user's field of vision while the user can see the real world, so as to mix the real world and the virtual world, allowing users to see the effect of augmented reality more intuitively.
[0003] With the continuous development of computer software and hardware, extended reality (XR) devices that can provide VR technology and / or AR technology are becoming more and more popular in family life. When multiple family members use the same head-mounted display device, the lack of privacy protection in the head-mounted display device is an urgent problem that needs to be solved. Summary of the invention
[0004] The embodiments of the present application provide an information interaction method and a head-mounted display device, which are used to protect the privacy of users in the head-mounted display device.
[0005] In a first aspect, an embodiment of the present application provides an information interaction method, the method comprising: a head-mounted display device obtains a first gesture image of a first user, and determines a first application space and a first response function corresponding to the first gesture image, wherein the first application space is an application space allocated by the head-mounted display device to the first user, and the first response function is a response made by the head-mounted display device to the first gesture image. In the first application space, the head-mounted display device executes the first response function. In this method, the first user can access the application software in the first application space, and at the same time, the first application space can be accessed by the first user, which can effectively protect the privacy of the first user.
[0006] In one possible design, the head-mounted display device determines the first application space corresponding to the first gesture image, which may include: the head-mounted display device inputs the first gesture image into the first identity re-identification model to obtain the first user name, and uses the application space corresponding to the first user name as the first application space, wherein the first identity re-identification model is obtained by training the first data set, and the first data set includes a preset gesture image with a preset user name label. With this design, the head-mounted display device inputs the first gesture image into the first identity re-identification model, and the first user name can be determined based on the first gesture image, which can effectively improve the recognition efficiency.
[0007] In a possible design, the head-mounted display device determines the first response function corresponding to the first gesture image, which may include: the head-mounted display device inputs the first gesture image into a gesture classification model to obtain the first response function, wherein the gesture classification model is obtained by training a second data set, and the second data set includes a preset gesture image with a preset response function label. Through this design, the head-mounted display device inputs the first gesture image into the gesture classification model, and determines the first response function according to the first gesture image, providing a convenient method for determining the first response function.
[0008] In a possible design, after the head-mounted display device obtains the first gesture image of the first user, it may also include: obtaining the first iris image of the first user; the head-mounted display device determines the first application space corresponding to the first gesture image, which may include: the head-mounted display device inputs the first gesture image and the first iris image into the second identity re-identification model, obtains the first user name, and uses the application space corresponding to the first user name as the first application space, wherein the second identity re-identification model is obtained by training a third data set, and the third data set includes a preset gesture image with a preset user name label, and a preset iris image with a preset user name label. With this design, the head-mounted display device inputs the first gesture image and the first iris image into the second identity re-identification model, and determines the first user name based on the first gesture image and the first iris image, which can effectively improve the recognition accuracy.
[0009] In a possible design, the method further includes: the head display device adds the first gesture image with the first user name label to the first data set, and uses the updated first data set to train the first identity re-identification model. Through this design, when the gesture made by the user is not standard, or when the texture information of the user's hand changes as the user ages, the first identity re-identification model can still perform accurate recognition.
[0010] In a possible design, the head-mounted display device adds the first gesture image with the first user name tag to the first data set, which may include: the head-mounted display device determines at least one preset gesture image with the first user name tag from the first data set, and then determines a first similarity between at least one preset gesture image with the first user name tag and the first gesture image, and then, when the first similarity is less than a first threshold and greater than a second threshold, the head-mounted display device adds the first gesture image with the first user name tag to the first data set. Through this design, a method for updating the first data set is provided.
[0011] In a possible design, the method further includes: the head display device adds the first gesture image with the first response function label to the second data set, and uses the updated second data set to train the gesture classification model. Through this design, when the gesture made by the user is not standard, or when the texture information of the user's hand changes as the user ages, the gesture classification model can still accurately identify it.
[0012] In a possible design, the head-mounted display device adds the first gesture image with the first response function label to the second data set, which may include: the head-mounted display device determines at least one preset gesture image with the first response function label from the second data set, and then determines the second similarity between the at least one preset gesture image with the first response function label and the first gesture image, and then, when the second similarity is less than the third threshold and greater than the fourth threshold, the head-mounted display device adds the first gesture image with the first response function label to the second data set. Through this design, a method for updating the second data set is provided.
[0013] In a possible design, the method further includes: the head display device adds the first gesture image with the first user name label and the first iris image with the first user name label to a third data set, and uses the updated third data set to train the second identity re-identification model. With this design, when the gesture made by the user is not standard, or when the user's hand texture information changes as the user ages, or when the user's iris changes, the second identity re-identification model can still perform accurate recognition.
[0014] In a possible design, the head display device adds the first gesture image with the first user name tag to the third data set, which may include: the head display device determines at least one preset gesture image with the first user name tag from the third data set, and then determines a third similarity between at least one preset gesture image with the first user name tag and the first gesture image, and then, when the third similarity is less than a fifth threshold and greater than a sixth threshold, the head display device adds the first gesture image with the first user name tag to the third data set. Through this design, a method for updating the third data set is provided.
[0015] In a second aspect, the present application further provides a head-mounted display device, which can implement the method in the first aspect and any possible implementation of the first aspect. The head-mounted display device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more units or modules corresponding to the above functions.
[0016] In a third aspect, an embodiment of the present application further provides a head-mounted display device, which includes a module / unit for executing the method in the first aspect and any possible implementation of the first aspect. These modules / units can be implemented by hardware, or corresponding software can be implemented by hardware.
[0017] In a fourth aspect, an embodiment of the present application provides a head-mounted display device, comprising a memory and a processor; wherein the processor is used to execute a computer program or instruction stored in the memory, so that the head-mounted display device implements the method in the above-mentioned first aspect and any possible implementation manner of the first aspect.
[0018] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium comprising a computer program. When the computer program runs on a head-mounted display device, the computer-readable storage medium implements the method in the above-mentioned first aspect and any possible implementation manner of the first aspect.
[0019] In a sixth aspect, an embodiment of the present application further provides a computer program product. When the computer program product is run on a head-mounted display device, the head-mounted display device implements the method in the above-mentioned first aspect and any possible implementation manner of the first aspect.
[0020] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1a A schematic diagram of the hardware structure of a head-mounted display device provided in an embodiment of the present application;
[0022] Figure 1b A schematic diagram of the hardware structure of a head-mounted display device provided in an embodiment of the present application;
[0023] Figure 1c A schematic diagram of the software structure of a head-mounted display device provided in an embodiment of the present application;
[0024] Figure 2 A flowchart of an information interaction method provided in an embodiment of the present application;
[0025] Figure 3 A schematic diagram of the structure of a first user interface provided in an embodiment of the present application;
[0026] Figure 4 A schematic diagram of the structure of a first user interface provided in an embodiment of the present application;
[0027] Figure 5 A schematic diagram of the structure of a first user interface provided in an embodiment of the present application;
[0028] Figure 6A schematic diagram of a preset gesture image provided in an embodiment of the present application;
[0029] Figure 7 A schematic diagram of a preset gesture image provided in an embodiment of the present application;
[0030] Figure 8 A schematic diagram of a preset gesture image provided in an embodiment of the present application;
[0031] Fig. 9 A schematic diagram of the structure of a gesture joint point provided in an embodiment of the present application;
[0032] Fig.10 A schematic diagram of the structure of a call interface provided in an embodiment of the present application;
[0033] Fig.11 A schematic diagram of a preset gesture image provided in an embodiment of the present application;
[0034] Fig.12 A schematic diagram of a preset gesture image provided in an embodiment of the present application;
[0035] Fig.13 A flowchart of an information interaction method provided in an embodiment of the present application;
[0036] Fig.14 A schematic diagram of a first gesture image provided in an embodiment of the present application;
[0037] Fig.15 A schematic diagram of the hardware structure of a head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0039] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.
[0040] XR devices can be VR devices or AR devices. VR devices can be position trackers, data gloves, head-mounted devices, etc., and AR devices can be mobile phones, tablets, laptops, or wearable devices with wireless communication functions (such as smart watches or smart glasses, etc.).
[0041] The XR device includes a device capable of implementing data processing functions (such as a processor, or an application processor, or an image processor, or other processor). Exemplary embodiments of the XR device include but are not limited to a device equipped with Or devices with other operating systems. The above-mentioned XR device may also be other portable devices, such as a laptop computer with a touch-sensitive surface (eg, a touch panel).
[0042] In conjunction with the accompanying drawings, taking the XR device as a head-mounted display device (referred to as a head-mounted display device) as an example, the structure of the above-mentioned XR device is further explained.
[0043] Figure 1a A schematic diagram of the structure of a head display device 100 and a schematic diagram of the structure of a handle 101 provided in an embodiment of the present application are shown. Figure 1a Based on the head display device 100 shown, the head display device may also have other variant structures, which are not limited here. The head display device 100 may be used by a user through Figure 1a The control may be performed by the handle 101 in the figure, or directly by the user's fingers, or by other devices, which are not limited here.
[0044] The hardware structure diagram corresponding to the head display device 100 is as follows: Figure 1b As shown, the head display device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0045] It is understandable that the structure shown in the embodiment of the present application does not constitute a specific limitation on the head display 100. In other embodiments of the present application, the head display device 100 may include Figure 1b More or fewer components than those shown in the drawings may be used, some components may be combined, some components may be separated, or different component arrangements may be used. Figure 1b The components shown may be implemented in hardware, software, or a combination of software and hardware.
[0046] Below Figure 1b The components of the head display device 100 shown are described in detail.
[0047] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may be the nerve center and command center of the head display device 100. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.
[0048] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory, thereby avoiding repeated access, reducing the waiting time of the processor 110, and thus improving the efficiency of the system.
[0049] In some embodiments, the processor 110 may include one or more interfaces. For example, the interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0050] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the head display device 100.
[0051] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.
[0052] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0053] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.
[0054] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the head display device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the head display device 100.
[0055] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0056] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the head display device 100, and can also be used to transmit data between the head display device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as mobile phones, etc.
[0057] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the head display device 100. In other embodiments of the present application, the head display device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0058] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the head display device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.
[0059] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0060] The wireless communication function of the head display device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.
[0061] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the head display device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0062] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the head display device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
[0063] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.
[0064] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the head display device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering processing, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0065] In some embodiments, the antenna 1 of the head display device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the head display device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).
[0066] The head display device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
[0067] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the head display device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0068] The head display device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0069] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
[0070] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the head display device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0071] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the head display device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0072] Video codecs are used to compress or decompress digital videos. The head mounted display device 100 may support one or more video codecs. Thus, the head mounted display device 100 may play or record videos in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0073] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, intelligent cognition and other applications of the head display device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0074] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the head display device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.
[0075] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the head display device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the head display device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.
[0076] The head display device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.
[0077] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.
[0078] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The head display device 100 can listen to music or listen to hands-free calls through the speaker 170A.
[0079] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the head display device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.
[0080] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The head display 100 can be provided with at least one microphone 170C. In other embodiments, the head display device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the head display device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.
[0081] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0082] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The head display device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the head display device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The head display device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.
[0083] The gyroscope sensor 180B can be used to determine the motion posture of the head display device 100. In some embodiments, the angular velocity of the head display device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for shooting anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.
[0084] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the head display device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
[0085] The magnetic sensor 180D includes a Hall sensor. The head display device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the head display device 100 is a flip-type head display, the head display device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.
[0086] The acceleration sensor 180E can detect the magnitude of the acceleration of the head display device 100 in various directions (generally three axes). When the head display device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching, pedometers, etc.
[0087] The distance sensor 180F is used to measure the distance. The head display device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the head display device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.
[0088] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The head-mounted display device 100 emits infrared light outward through the light emitting diode. The head-mounted display device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the head-mounted display 100. When insufficient reflected light is detected, the head-mounted display device 100 can determine that there is no object near the head-mounted display device 100. The head-mounted display device 100 can use the proximity light sensor 180G to detect that the user holds the head-mounted display device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.
[0089] The ambient light sensor 180L is used to sense the brightness of the ambient light. The head display device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived brightness of the ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the head display device 100 is in a packaging bag to prevent accidental touch.
[0090] The fingerprint sensor 180H is used to collect fingerprints. The head display device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc. For example, a fingerprint sensor can be configured on the front of the head display device 100 (below the display screen 194), or a fingerprint sensor can be configured on the back of the head display device 100 (below the rear camera). In addition, the fingerprint recognition function can also be realized by configuring a fingerprint sensor in the touch screen, that is, the fingerprint sensor can be integrated with the touch screen to realize the fingerprint recognition function of the head display device 100. In this case, the fingerprint sensor can be configured in the touch screen, can be a part of the touch screen, or can be configured in the touch screen in other ways. In addition, the fingerprint sensor can also be implemented as a full-panel fingerprint sensor, so the touch screen can be regarded as a panel where fingerprint collection can be performed at any position. In some embodiments, the fingerprint sensor can process the collected fingerprint (for example, whether the fingerprint is verified) and send it to the processor 110, and the processor 110 makes corresponding processing according to the fingerprint processing result. In other embodiments, the fingerprint sensor may also send the collected fingerprint to the processor 110 so that the processor 110 processes the fingerprint (e.g., fingerprint verification, etc.). The fingerprint sensor in the embodiment of the present application may adopt any type of sensing technology, including but not limited to optical, capacitive, piezoelectric or ultrasonic sensing technology, etc.
[0091] The temperature sensor 180J is used to detect temperature. In some embodiments, the head display device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the head display device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the head display device 100 heats the battery 142 to prevent the head display device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the head display device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.
[0092] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the head display device 100, which is different from the position of the display screen 194.
[0093] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.
[0094] The button 190 includes a power button, a volume button, etc. The button 190 may be a mechanical button or a touch button. The head display device 100 may receive a button input and generate a key signal input related to the user setting and function control of the head display device 100.
[0095] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0096] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.
[0097] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the head-mounted display device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The head-mounted display device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The head-mounted display device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the head-mounted display device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the head-mounted display device 100 and cannot be separated from the head-mounted display device 100.
[0098] although Figure 1b Not shown, the head display device 100 may also include a Bluetooth device, a positioning device, a flash light, a micro-projection device, a near field communication (NFC) device, etc., which are not described in detail here.
[0099] The software system of the head display device 100 may adopt a layered architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the head display device 100.
[0100] Figure 1c 1 is a software structure block diagram of the head display device 100 according to an embodiment of the present invention.
[0101] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.
[0102] The application layer can include a series of application packages.
[0103] like Figure 1c As shown, the application package may include applications such as phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.
[0104] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0105] like Figure 1c As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.
[0106] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
[0107] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0108] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
[0109] The phone manager is used to provide communication functions for the head display device 100, such as the management of call status (including connecting, hanging up, etc.).
[0110] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.
[0111] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0112] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.
[0113] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
[0114] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0115] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0116] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0117] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0118] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0119] A 2D graphics engine is a drawing engine for 2D drawings.
[0120] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, sensor driver, audio driver, and microphone driver.
[0121] The following embodiments can all be implemented in an electronic device having the above hardware structure (such as a head display device 100). The following embodiments will take the head display device 100 as an example and describe in detail the information interaction method provided in the embodiments of the present application in conjunction with the accompanying drawings.
[0122] Figure 2 The following is a flow chart showing an information interaction method provided by an embodiment of the present application. Figure 1a The method flow can be implemented by the head display device alone, or can also be implemented through data interaction between the mobile terminal and the head display device. The information interaction method may include:
[0123] S201, the head display device enters a preset gesture image entry state, wherein the preset gesture in the preset gesture image can be any user-defined gesture.
[0124] In the embodiment of the present application, the head display device can enter the preset gesture image input state in the following manner:
[0125] In some implementations, the head-mounted display device may receive a first voice message, and determine, based on the first voice message, whether to trigger the head-mounted display device to enter a preset gesture image entry state.
[0126] In some embodiments, the head-mounted display device can directly receive the user's voice message as the first voice message. For example, within the distance range where the head-mounted display device can receive voice messages, the user says "custom gesture entry", the head-mounted display device receives the voice message, and uses "custom gesture entry" as the first voice message.
[0127] In some embodiments, the head-mounted display device can receive a first voice message sent from a mobile terminal. For example, within the distance range where the mobile terminal can receive voice messages, the user says "custom gesture entry", and after the mobile terminal receives the voice message, it sends "custom gesture entry" to the head-mounted display device, and the head-mounted display device receives "custom gesture entry" as the first voice message. In the embodiments of the present application, the mobile terminal can be a mobile phone, etc., which is not limited here.
[0128] After the head-mounted display device obtains the first voice message based on any of the above embodiments, it can perform voice recognition on the first voice message to determine whether the first voice message contains the first preset word. If it contains the first preset word, the head-mounted display device is triggered to enter the preset gesture image entry state; if it does not contain the first preset word, the head-mounted display device does not perform any processing.
[0129] The first preset word may include a first gesture type or a second gesture type. The first gesture type may be a gesture type corresponding to any specific function, for example, the first gesture type may be a gallery opening gesture, a call answering gesture, a gallery gesture, an incoming call gesture, etc. The second gesture type may be a gesture type corresponding to a non-specific function, for example, the second gesture type may be a custom gesture, etc.
[0130] For example, if the first voice message is “Gallery gesture entry”, the head display device performs voice recognition on “Gallery gesture entry” and determines that “Gallery gesture entry” contains the first preset word “Gallery gesture”, the head display device can display the following information: Figure 3 The first user interface shown. The first user interface includes 4 virtual buttons, namely: gallery gesture, open gallery gesture, close gallery gesture, and custom operation gesture. Among them, the "gallery gesture" virtual button is used to identify the gesture type, the "open gallery gesture" virtual button is used to obtain the user's open gallery gesture, the "close gallery gesture" virtual button is used to obtain the user's close gallery gesture, and the "custom operation gesture" virtual button is used to obtain the gesture corresponding to any user-defined response function, for example, a user-defined gallery clearing gesture.
[0131] If the first voice message is “custom gesture entry”, the head display device performs voice recognition on “custom gesture entry” and determines that “custom gesture entry” contains the first preset word “custom gesture”, and the head display device displays the following: Figure 4 The second user interface shown. The second user interface includes 4 virtual buttons, namely: custom gesture, gallery gesture, incoming call gesture, and other application gestures. Among them, the "custom gesture" virtual button is used to identify the gesture type, the "gallery gesture" virtual button is used to obtain gestures related to the gallery, the "incoming call gesture" virtual button is used to obtain gestures related to incoming calls, and the "other application gesture" virtual button is used to obtain gestures related to other applications. Other applications can be any application installed in the head display device.
[0132] It should be understood that the first preset word in the head display device can be determined by the device manufacturer or by the user. The first preset word in the head display device can be fixed or continuously updated.
[0133] In some implementations, the head-mounted display device may receive a first keyboard message, where the first keyboard message is used to trigger the head-mounted display device to enter a preset gesture image entry state.
[0134] In some embodiments, the head display device may directly receive a first keyboard message from the user. For example, the user clicks a first key in a virtual keyboard of the head display device, and the head display device generates a first keyboard message in response to the click operation on the first key.
[0135] In some embodiments, the head display device may receive a first keyboard message from the mobile terminal. For example, a user clicks a first key on a keyboard of the mobile terminal, and the mobile terminal generates a first keyboard message in response to the click operation on the first key, and sends the first keyboard message to the head display device.
[0136] It should be understood that the first button may be a virtual button on a virtual keyboard of a head-mounted display device, or may be a physical button on a physical keyboard of a mobile terminal.
[0137] In the embodiment of the present application, the first button is a virtual button on the virtual keyboard of the head display device as an example for explanation. The first button can be a virtual button corresponding to the first gesture type, or a virtual button corresponding to the second gesture type. The first gesture type can be any specific gesture type, for example, the first gesture type can be a gallery opening gesture, a call answering gesture, a gallery gesture, an incoming call gesture, etc. The second gesture type can be a non-specific gesture type, for example, the second gesture type can be a custom gesture, etc.
[0138] For example, if the first button is a “Gallery Gesture” virtual button, after the user clicks the “Gallery Gesture” virtual button, the head display device obtains the first keyboard message, and the head display device displays the following: Figure 3 The first user interface is shown.
[0139] If the first button is a "custom gesture" virtual button, after the user clicks the "custom gesture" virtual button, the head display device obtains the first keyboard message, and the head display device displays the following Figure 4 The first user interface is shown.
[0140] It should be understood that before the mobile terminal sends the first voice message or the first keyboard message to the head display device, the mobile terminal can establish a connection with the head display device, and the connection can be a wireless connection or a wired connection. If the mobile terminal and the head display device are wirelessly connected, it can be understood that the wireless connection between the mobile terminal and the head display device is achieved through the wireless communication module in the mobile terminal and the wireless communication module in the head display device. For example, the mobile terminal and the head display device can be wirelessly connected via WIFI or via Bluetooth.
[0141] S202, the head display device obtains a preset gesture image, a preset response function, and a preset user name, and generates a first data set according to the preset gesture image and the preset user name, and generates a second data set according to the preset gesture image and the preset response function.
[0142] In an embodiment of the present application, the preset gesture image is determined based on a gesture made by a user corresponding to a preset user name. The preset gesture image corresponds to a preset response function, and the preset response function is the response made by the head display device to the preset gesture image.
[0143] When the head display device displays the first user interface, Figure 5 The first user interface shown in the figure is used as an example for explanation. Figure 5 As shown in the figure, if the user clicks the "Open Gallery Gesture" virtual button, the head display device starts the camera on the head display device and obtains the user's gesture through the camera. The preset response function corresponding to the "Open Gallery Gesture" virtual button is to open the gallery, and the image including the user's gesture obtained by the camera is used as the preset gesture image. Figure 6 , then Figure 6 As the preset gesture image corresponding to the preset response function "Open Gallery", and the Figure 6 Add to the second dataset.
[0144] For other virtual buttons of the first user interface, other preset gesture images can also be obtained. The method for obtaining the preset gesture image is similar to the method for obtaining the preset gesture image for the above-mentioned "open gallery gesture" virtual button, and is not limited here.
[0145] It should be understood that in order to ensure the accuracy of subsequent recognition, the preset gesture image corresponding to the preset response function obtained by the head-mounted display device can be a single preset gesture image or multiple preset gesture images of the same user, wherein the gestures in the multiple preset gesture images are relatively similar, but there are some differences.
[0146] Take the head display device to obtain two preset gesture images as an example. The preset gesture images corresponding to the opening gallery function include: Figure 7 and Figure 8 . Figure 7 It includes three joint points, namely joint point A, joint point B and joint point C. Figure 8 It includes three joint points, namely joint point E, joint point F and joint point G. Figure 7 The line segment between joint point A and joint point B is AB, the line segment between joint point B and joint point C is BC, and the angle between line segment AB and line segment BC is angle 1. Figure 8The line segment between joint point E and joint point F is EF, the line segment between joint point F and joint point G is FG, and the angle between line segment EF and line segment FG is angle 2. It can be seen that the size of angle 1 is not equal to angle 2. Therefore, for the opening gallery function, the head display device obtains two preset gesture images. The gestures in the above two preset gesture images are similar, but there are some differences.
[0147] When the head display device displays the second user interface, Figure 4 The second user interface shown in FIG. 1 is used as an example to explain that if the user clicks the “Gallery Gesture” virtual button, the head display device displays Figure 3 The first user interface shown is for Figure 3 How to operate the first user interface shown has been described in detail above and will not be repeated here.
[0148] In the embodiment of the present application, the head display device also needs to obtain a preset user name. After obtaining the preset user name, it is determined whether the preset user name already exists. If the preset user name does not exist, a corresponding application space is allocated to the preset user name; if the preset user name already exists, the application space corresponding to the preset user name also exists, and there is no need to allocate an application space for the preset user name again. The application space allocated to the preset user name includes different application software installed on the head display device. The user can access the application space corresponding to the preset user name and use the application software in the application space according to the preset user name, and the application space can only be accessed by the user corresponding to the preset user name.
[0149] After obtaining the preset user name, the head display device adds the preset gesture image with the preset user name label to the first data set.
[0150] The head display device can obtain the preset user name through the following implementation method.
[0151] In some embodiments, before executing S202, the head display device receives a second voice message, wherein the second voice message includes a preset user name, and the head display device performs voice recognition on the second voice message to obtain the preset user name.
[0152] Alternatively, the first voice message received by the head display device includes a preset user name, and the head display device performs voice recognition on the first voice message to obtain the preset user name.
[0153] In some embodiments, before executing S202, the head display device receives a second keyboard message, and the head display device obtains a preset user name based on the second keyboard message. For example, the user name input by the user through the virtual keyboard of the head display device is used as the second keyboard message, and the head display device obtains the user name as the preset user name based on the second keyboard message.
[0154] Optionally, when the head-mounted display device obtains the preset gesture image, the preset response function and the preset user name, the head-mounted display device may also collect the iris of the user corresponding to the preset user name, and generate a preset iris image based on the collected iris.
[0155] It should be understood that after the user puts on the head-mounted display device, an iris collector is installed on the inside of the head-mounted display device, which can directly collect the user's iris.
[0156] After acquiring the preset gesture image, the preset response function, the preset user name, and the preset iris image, the head display device generates a third data set according to the preset gesture image, the preset iris image, and the preset user name. Specifically, the head display device adds the preset gesture image with the preset user name tag and the preset iris image with the preset user name tag as a whole to the third data set.
[0157] S203: The head mounted display device stores the first data set and the second data set.
[0158] The head-mounted display device can store the first data set and the second data set in the head-mounted display device, and can also store them in a cloud server.
[0159] Optionally, after storing the first data set and the second data set, the head display device may perform the following two processing implementations:
[0160] In some embodiments, the head-mounted display device trains a first identity re-identification model based on the first data set, and updates a gesture classification model based on the second data set. Subsequently, the head-mounted display device can recognize the first gesture image of the first user based on the trained first identity re-identification model and gesture classification model.
[0161] It should be understood that when the head-mounted display device leaves the factory, the head-mounted display device may store the initialized first identity re-identification model and gesture classification model; the head-mounted display device may also not store the first identity re-identification model and gesture classification model, but download the initialized first identity re-identification model and gesture classification model from the cloud server when in use. The parameters in the initialized first identity re-identification model and gesture classification model may be random initial values, or may be obtained after training with other data sets.
[0162] Taking the head display device storing the initialized first identity re-identification model as an example, the head display device obtains a first data set and inputs the first data set into the first identity re-identification model so as to train the first identity re-identification model, wherein the preset gesture image in the first data set is used as the first training data, and the preset user name in the first data set is used as the first real label. Specifically, the first identity re-identification model includes three modules, namely, a first feature extraction module, a first classification module, and a first loss function module. During the training process of the above-mentioned first identity re-identification model, the first feature extraction module receives the preset gesture image, extracts features from the preset gesture image, obtains the first preset feature, and then outputs the first preset feature to the first classification module. The first classification module receives the first preset feature and outputs the first predicted label of the preset gesture image based on the first preset feature. The first loss function module receives the first predicted label and the first real label, calculates the difference between the first predicted label and the first real label, obtains the first loss value, and updates the parameters in the first feature extraction module, the first classification module, and the first loss function module in the first identity re-identification model based on the first loss value, thereby realizing the training of the first identity re-identification model. It should be understood that the first preset feature may include at least one of the following: gesture joints, hand texture information. Fig. 9 The 21 key points shown, the gesture joint points may also include other key points, which are not limited here. The hand texture information may be at least one information such as palm print, skin texture, etc., which are not limited here.
[0163] After the first identity re-identification model is trained, if the first gesture image of the first user is input into the first identity re-identification model, the first identity re-identification model can output a user name predicted for the first gesture image, and use the user name as the first user name.
[0164] Taking the gesture classification model initialized and stored in the head display device as an example, the head display device obtains a second data set and inputs the second data set into the gesture classification model to train the gesture classification model, wherein the preset gesture image in the second data set is used as the second training data, and the preset response function in the second data set is used as the second real label. Specifically, the gesture classification model includes three modules, namely a second feature extraction module, a second classification module, and a second loss function module. During the training process of the above-mentioned gesture classification model, the second feature extraction module receives the preset gesture image, performs feature extraction on the preset gesture image, obtains the second preset feature, and then outputs the second preset feature to the second classification module. The second classification model receives the second preset feature and outputs the second predicted label of the preset gesture image based on the second preset feature. The second loss function module receives the second predicted label and the second real label, calculates the difference between the second predicted label and the second real label, obtains the second loss value, and updates the parameters in the second feature extraction module, the second classification module, and the second loss function module in the gesture classification model based on the second loss value, so as to realize the training of the gesture classification model. It should be understood that the second preset feature may include at least one of the following: gesture joint points, hand texture information. Among them, the gesture joint points may include such as Fig. 9 The 21 key points shown, the gesture joint points may also include other key points, which are not limited here. The hand texture information may be at least one information such as palm print, skin texture, etc., which are not limited here.
[0165] After the gesture classification model is trained, if a first gesture image of a first user is input into the gesture classification model, the gesture classification model can output a response function predicted for the first gesture image, and use the response function as the first response function.
[0166] In an embodiment of the present application, different gesture classification models may be trained for different preset user names; or the same gesture classification model may be trained.
[0167] Optionally, for the third data set, the head display device may store the third data set in the head display device or in a cloud server. The head display device may also train a second identity re-identification model based on the third data set. Subsequently, the head display device may recognize the first gesture image of the first user based on the trained second identity re-identification model and the gesture classification model.
[0168] It should be understood that when the head-mounted display device leaves the factory, the head-mounted display device may store the initialized second identity re-identification model; the head-mounted display device may also not store the second identity re-identification model, but download the initialized second identity re-identification model from the cloud server when in use. The parameters in the initialized second identity re-identification model may be random initial values, or may be obtained after training with other data sets.
[0169] Taking the second identity re-identification model initialized by the head display device as an example, the head display device obtains a third data set and inputs the third data set into the second identity re-identification model so as to train the second identity re-identification model, wherein the preset gesture image and the preset iris image in the third data set are used as the third training data, and the preset user name in the third data set is used as the third real label. Specifically, the second identity re-identification model includes four modules, namely a third feature extraction module, a fourth feature extraction module, a third classification module and a third loss function module. In the above-mentioned second identity re-identification model training process, the third feature extraction module receives the preset gesture image, and performs feature extraction on the preset gesture image to obtain the third preset feature, and then outputs the third preset feature to the third classification module. The fourth feature extraction module receives the preset iris image, and performs feature extraction on the preset iris image to obtain the fourth preset feature, and then outputs the fourth preset feature to the third classification module. The third classification module receives the third preset feature and the fourth preset feature, and outputs the third predicted label of the preset gesture image based on the third preset feature and the fourth preset feature. The third loss function module receives the third predicted label and the third true label, calculates the difference between the third predicted label and the third true label, obtains a third loss value, and updates the parameters in the third feature extraction module, the fourth feature extraction module, the third classification module, and the third loss function module in the second identity re-identification model based on the third loss value, thereby implementing the training of the second identity re-identification model. It should be understood that the third preset feature may include at least one of the following: gesture joints, hand texture information. Among them, the gesture joints may include Fig. 9 The 21 key points shown, the gesture joint points may also include other key points, which are not limited here. The hand texture information may be at least one information such as palm print, skin texture, etc., which are not limited here.
[0170] After the second identity re-identification model is trained, if the first gesture image and the first iris image of the first user are input into the second identity re-identification model, the second identity re-identification model can output a user name predicted for the first gesture image and the first iris image as the first user name.
[0171] In some embodiments, the head display device uses a machine learning method to extract features from the preset gesture image to obtain a fifth preset feature. Subsequently, the head display device can recognize the first gesture image of the first user based on the fifth preset feature.
[0172] In the embodiment of the present application, the fifth preset feature may include at least one of the following: gesture joints, hand texture information. Fig. 9The 21 key points shown, the gesture joint points may also include other key points, which are not limited here. The hand texture information may be at least one information such as palm print, skin texture, etc., which are not limited here.
[0173] After acquiring the fifth preset feature, the head display device can associate the preset gesture image with the fifth preset feature to generate and store a fourth association relationship.
[0174] Optionally, the head display device may also use a machine learning method to extract features from the preset iris image to obtain a sixth preset feature. Subsequently, the head display device may recognize the first gesture image of the first user based on the sixth preset feature, or based on the fifth preset feature and the sixth preset feature.
[0175] After acquiring the sixth preset feature, the head display device can associate the preset iris image with the sixth preset feature to generate and store a fifth association relationship.
[0176] In addition, the head display device may also associate the preset gesture image and the preset user name in the first data set to generate and store a first association relationship. The head display device may also associate the preset gesture image and the preset response function in the second data set to generate and store a second association relationship. The head display device may also associate the preset gesture image, the preset iris image, and the preset user name in the third data set to generate and store a third association relationship.
[0177] It should be understood that the machine learning method involved in the embodiments of the present application is not specifically limited.
[0178] When a user uses a head-mounted display device, if the user's mobile terminal receives an incoming call, the current conventional practice is for the user to remove the head-mounted display device and then answer the mobile terminal; if the user wants to use the mobile terminal to make a call, the user needs to remove the head-mounted display device and then dial the number on the mobile terminal. In order to allow users to conveniently answer or make calls, the head-mounted display device provided in the embodiment of the present application has a mobile communication function. In order to achieve this function, the embodiment of the present application provides the following two possible implementation methods.
[0179] In some embodiments, the head-mounted display device is connected to a mobile terminal, and the head-mounted display device can directly and remotely use the communication function of the mobile terminal.
[0180] It should be understood that the head-mounted display device and the mobile terminal can be connected by wire or by wireless connection. If the mobile terminal and the head-mounted display device are connected wirelessly, it can be understood that the wireless connection between the mobile terminal and the head-mounted display device is achieved through the wireless communication module in the mobile terminal and the wireless communication module in the head-mounted display device. For example, the mobile terminal and the head-mounted display device can be wirelessly connected via WIFI or via Bluetooth.
[0181] After the head-mounted display device is connected to the mobile terminal, the head-mounted display device can use the communication function of the mobile terminal. For example, if the mobile terminal has an incoming call prompt, the mobile terminal and the head-mounted display device can both display the following information: Fig.10 The user can click the "Accept" virtual button in the call interface displayed on the head-mounted display device to communicate without removing the head-mounted display device, or click the "Reject" virtual button to reject the communication.
[0182] In the embodiment of the present application, the user can click a virtual button on the call interface displayed by the head display device through the following possible implementation methods: Figure 1a The handle 101 in the figure can be operated directly by fingers or by preset gestures set by users.
[0183] Taking the user's operation through the preset gesture as an example, the call answering gesture corresponding to the "answering call" function is as follows: Fig.11 As shown in the figure, the call rejection gesture corresponding to the "Call Rejection" function is as follows Fig.12 When the head display device displays Fig.10 When the user makes a call in the visible area of the head display device, Fig.11 The gesture shown is equivalent to clicking the "Accept" virtual button on the call interface displayed on the head display device, and the mobile terminal can communicate; if the user makes a Fig.12 The gesture shown is equivalent to clicking the "Reject" virtual button on the call interface displayed on the head display device, and the mobile terminal rejects the communication. Regardless of whether the mobile terminal communicates or not, the above call records can be saved in the head display device or not.
[0184] Optionally, the address book, call logs, etc. in the mobile terminal can be uploaded to the head-mounted display device for storage in the head-mounted display device. The user can also manually update the address book through the virtual keyboard of the head-mounted display device. For example, the user can enter the contact name and contact mobile phone number in the head-mounted display device to update the address book.
[0185] In some embodiments, the head mounted device includes Figure 1b The mobile communication module 150 shown in the figure enables the head display device to have a communication function. A SIM card can be inserted into the SIM card interface 195 of the head display device, so that the head display device has a dialable number, wherein the number corresponding to the SIM card in the head display device and the number corresponding to the SIM card in the mobile terminal can be the same or different.
[0186] For example, if the head-mounted display device has an incoming call notification, the head-mounted display device can display the following Fig.10In the call interface shown, the user can click the "Accept" virtual button in the call interface displayed on the head display device to communicate, or click the "Reject" virtual button to reject the communication.
[0187] Optionally, the head-mounted display device can also be connected to a mobile terminal, wherein the head-mounted display device and the mobile terminal can be connected by wire or by wireless connection. After the head-mounted display device and the mobile terminal are connected, the address book, call records, etc. in the mobile terminal can be uploaded to the head-mounted display device for storage in the head-mounted display device. The user can also manually update the address book through the virtual keyboard of the head-mounted display device. For example, the user can enter the contact name and contact mobile phone number in the head-mounted display device to update the address book.
[0188] Through the above method, when the user is using the head display device, if there is an incoming call prompt, the user does not need to remove the head display device, but can directly use the head display device to answer the call; similarly, if the user wants to make a call, the user does not need to remove the head display device, but can directly use the head display device to make a call. Therefore, the head display device can provide a convenient mobile communication function.
[0189] In order to protect the privacy of users using a head-mounted display device, an embodiment of the present application provides an information interaction method, that is, when a first user uses a head-mounted display device, the head-mounted display device displays a first application space corresponding to the first user, so that the first user can access the first application space and use the application software in the first application space. At the same time, in order to improve the convenience of using the head-mounted display device, the first user can conveniently use the application software in the first application space through customized gestures, wherein the first user can be a user corresponding to any preset user name. Therefore, when the first user uses the head-mounted display device, the head-mounted display device can execute the following Fig.13 The following steps are shown:
[0190] S1301: The head display device obtains a first gesture image of a first user, and determines a first application space and a first response function corresponding to the first gesture image.
[0191] The first gesture image may include one gesture image or multiple gesture images, which is not limited here. The first application space is the application space allocated by the head display device to the first user, and the first response function is the response made by the head display device to the first gesture image.
[0192] The head display device can determine the first application space and the first response function through the following possible implementations.
[0193] In an embodiment of the present application, the head display device can determine the first application space based on the first identity re-identification model, and determine the first response function based on the gesture classification model.
[0194] The head-mounted display device determines the first application space based on the first identity re-identification model, which can include the following steps: the head-mounted display device first determines the first user name corresponding to the first gesture image based on the first identity re-identification model, and uses the application space corresponding to the first user name as the first application space, wherein the first identity re-identification model can be obtained by training with the first data set.
[0195] For example, if the first identity re-identification model is obtained based on the training of the first data set, the head display device inputs the first gesture image into the first identity re-identification model after obtaining the first gesture image of the first user, wherein the first identity re-identification model includes three modules, namely, a first feature extraction module, a first classification module and a first loss function module, and the first loss function module is only used when training the first identity re-identification model. The first feature extraction module receives the first gesture image, extracts features from the first gesture image, obtains the first feature, and then outputs the first feature to the first classification module. The first classification module receives the first feature, and outputs the fourth predicted label of the first gesture image based on the first feature, and the fourth predicted label is the first user name.
[0196] It should be understood that the first feature may include at least one of the following: a gesture joint point, hand texture information. The first feature corresponds to the first preset feature, for example, if the first preset feature is a gesture joint point, the first feature is also a gesture joint point.
[0197] Through the above method, the head display device inputs the first gesture image into the first identity re-identification model, and the first user name can be determined only based on the first gesture image, which can effectively improve the recognition efficiency.
[0198] The head-mounted display device can also determine the first application space based on the second identity re-identification model, which can include the following steps: the head-mounted display device determines the first user name corresponding to the first gesture image based on the second identity re-identification model, and uses the application space corresponding to the first user name as the first application space, wherein the second identity re-identification model can be obtained through training with a third data set.
[0199] For example, if the second identity re-identification model is obtained based on the third data set training, then after obtaining the first gesture image of the first user, the head display device can also obtain the iris image of the first user as the first iris image, and the head display device inputs the first gesture image and the first iris image into the second identity re-identification model, wherein the second identity re-identification model includes four modules, namely, a third feature extraction module, a fourth feature extraction module, a third classification module and a third loss function module, and the third loss function module is only used when training the second identity re-identification model. The third feature extraction module receives the first gesture image, and performs feature extraction on the first gesture image to obtain the third feature, and then outputs the third feature to the third classification module. The fourth feature extraction module receives the first iris image, and performs feature extraction on the first iris image to obtain the fourth feature, and then outputs the fourth feature to the third classification module. The third classification module receives the third feature and the fourth feature, and outputs the fifth preset label of the first gesture image based on the third feature and the fourth feature, and the fifth predicted label is the first user name.
[0200] It should be understood that the third feature may include at least one of the following: gesture joints, hand texture information. The third feature corresponds to the third preset feature. For example, if the third preset feature is a gesture joint, the third feature is also a gesture joint.
[0201] Through the above method, the head display device inputs the first gesture image and the first iris image into the second identity re-identification model, and determines the first user name according to the first gesture image and the first iris image, which can effectively improve the recognition accuracy.
[0202] The head-mounted display device determines the first response function based on the gesture classification model, which may include the following steps: the head-mounted display device determines the first response function corresponding to the first gesture image based on the gesture classification model, wherein the gesture classification model may be obtained through training with the second data set.
[0203] For example, if the gesture classification model is obtained based on the training of the second data set, the head display device inputs the first gesture image into the gesture classification model after obtaining the first gesture image of the first user, wherein the gesture classification model includes three modules, namely, a second feature extraction module, a second classification module and a second loss function module, and the second loss function module is only used when training the gesture classification model. The second feature extraction module receives the first gesture image, extracts features from the first gesture image, obtains the second feature, and then outputs the second feature to the second classification module. The second classification module receives the second feature, and outputs the sixth predicted label of the first gesture image based on the second feature, and the sixth predicted label is the first response function.
[0204] It should be understood that the second feature may include at least one of the following: a gesture joint point, hand texture information. The second feature corresponds to the second preset feature, for example, if the second preset feature is a gesture joint point, the second feature is also a gesture joint point.
[0205] Through the above method, the head display device inputs the first gesture image into the gesture classification model, and determines the first response function according to the first gesture image, thereby providing a convenient method for determining the first response function.
[0206] In the embodiments of the present application, as the user becomes more and more proficient in using the head-mounted display device, the gestures presented when using the head-mounted display device will become relatively less and less standard; or, as the user ages, the user's hand texture information will gradually change. In order to effectively ensure the accuracy of the first identity re-identification model, the gesture classification model, and the second identity re-identification model, the above models need to be updated regularly.
[0207] After determining the first user name and the first response function corresponding to the first gesture image, the head display device can update the first identity re-identification model, the gesture classification model and the second identity re-identification model according to the first gesture image, the first iris image, the first user name and the first response function.
[0208] In some embodiments, the head-mounted display device can directly add the first gesture image with the first username label to the first data set, and use the updated first data to train the first identity re-identification model.
[0209] In some embodiments, the head display device may also determine at least one preset gesture image with a first user name tag from the first data set, and use the determined preset gesture image with the first user name tag as a first candidate gesture image. Then, a first similarity between the at least one first candidate gesture image and the first gesture image is determined, and the following possible implementations may be performed according to the first similarity:
[0210] In some embodiments, when the first similarity is less than the first threshold, the head display device adds the first gesture image with the first user name label to the first data set, and uses the updated first data set to train the first identity re-identification model. When the first similarity is greater than or equal to the first threshold, the head display device does not perform other processing.
[0211] The updated first data set obtained by the above implementation method does not include gesture images with a large similarity, so when the updated first data set is used to train the first identity re-identification model, it is beneficial to improve the accuracy of the first identity re-identification model.
[0212] In some embodiments, when the first similarity is less than the first threshold and greater than the second threshold, the head display device adds the first gesture image with the first user name label to the first data set, and uses the updated first data set to train the first identity re-identification model. When the first similarity is greater than or equal to the first threshold, or when the first similarity is less than or equal to the second threshold, the head display device does not perform other processing. The second threshold is less than the first threshold.
[0213] In the above method, if the first similarity is greater than or equal to the first threshold, it means that the first gesture image and at least one first candidate gesture image have a high similarity, and the first gesture image with the first user name label may not be used to update the first data set. If the first gesture image with the first user name label is added to the first data set, since the first gesture image and at least one first candidate gesture image have a high similarity, using more similar gesture images to train the first identity re-identification model may cause the first identity re-identification model to overfit, thereby affecting the accuracy of the first identity re-identification model.
[0214] If the first similarity is less than or equal to the second threshold, it means that the similarity between the first gesture image and at least one first candidate gesture image is small, and the first gesture image with the first user name label may not be used to update the first data set. If the first gesture image with the first user name label is added to the first data set, since the similarity between the first gesture image and at least one first candidate gesture image is small, the first gesture image is equivalent to noise data relative to other first candidate gesture images, which may affect the subsequent training of the first identity re-identification model, and further affect the accuracy of the first identity re-identification model.
[0215] The updated first data set obtained by the above implementation does not include gesture images with a large similarity and noise data. Therefore, when the updated first data set is used to train the first identity re-identification model, it is more conducive to improving the accuracy of the first identity re-identification model.
[0216] In some embodiments, when the first similarity is less than the first threshold, or when the first similarity is less than the first threshold and greater than the second threshold, before the head display device adds the first gesture image with the first user name tag to the first data set, the head display device can determine whether the number of first candidate gesture images is less than the first number threshold, if the number of first candidate gesture images is less than the first number threshold, the head display device adds the first gesture image with the first user name tag to the first data set; if the number of first candidate gesture images is greater than or equal to the first number threshold, the head display device deletes a preset gesture image with the first user name tag from the first data set, and adds the first gesture image with the first user name tag to the first data set, wherein the deleted preset gesture image with the first user name tag can be randomly selected or can be the one with the least similarity to the first gesture image. Through the above method, the storage space occupied by the first data set can be limited, and if the preset gesture image with the least similarity to the first gesture image is deleted from the first data set, it is equivalent to removing noise data in the first data set, which is conducive to improving the accuracy of the first identity re-identification model.
[0217] In an embodiment of the present application, the first similarity may be determined in the following possible ways: the head-mounted display device determines the similarity between each first candidate gesture image and the first gesture image respectively, and takes the average of the determined similarities as the first similarity; or, the head-mounted display device determines the similarity between each first candidate gesture image and the first gesture image respectively, and takes the determined minimum similarity as the first similarity.
[0218] It should be understood that when calculating the similarity between the first candidate gesture image and the first gesture image, the similarity can be calculated based on the features of the first candidate gesture image and the features of the first gesture image. Among them, the features of the first candidate gesture image and the features of the first gesture image can include at least one of the following: gesture joints, hand texture information. The features of the first candidate gesture image correspond to the features of the first gesture image. For example, if the features of the first candidate gesture image are gesture joints, the features of the first gesture image are also gesture joints.
[0219] Through the above method, when the gestures made by the user are not standard, or when the texture information of the user's hand changes as the user ages, the first identity re-identification model can still perform accurate identification.
[0220] In some embodiments, the head-mounted display device can directly add the first gesture image with the first response function label to the second data set, and use the updated second data set to train the gesture classification model.
[0221] In some embodiments, the head display device may also determine at least one preset gesture image with a first response function label from the second data set, and use the determined preset gesture image with the first response function label as the second candidate gesture image. Then, a second similarity between the at least one second candidate gesture image and the first gesture image is determined, and the following possible implementations may be performed according to the second similarity:
[0222] In some embodiments, when the second similarity is less than the third threshold, the head display device adds the first gesture image with the first response function label to the second data set, and uses the updated second data set to train the gesture classification model. When the second similarity is greater than or equal to the third threshold, the head display device does not perform other processing.
[0223] In some embodiments, when the second similarity is less than the third threshold and greater than the fourth threshold, the head display device adds the first gesture image with the first response function label to the second data set, and uses the updated second data set to train the gesture classification model. When the second similarity is greater than or equal to the third threshold, or when the second similarity is less than or equal to the fourth threshold, the head display device does not perform other processing. The fourth threshold is less than the third threshold.
[0224] The above method of obtaining the updated second data set is conducive to improving the accuracy of the gesture classification model. The specific analysis process can refer to the method of obtaining the updated first data set, which will not be described in detail here.
[0225] In some embodiments, when the second similarity is less than the third threshold, or when the second similarity is less than the third threshold and greater than the fourth threshold, before the head-mounted display device adds the first gesture image with the first response function label to the second data set, the head-mounted display device can determine whether the number of second candidate gesture images is less than the second number threshold, and if the number of second candidate gesture images is less than the second number threshold, the head-mounted display device adds the first gesture image with the first response function label to the second data set; if the number of second candidate gesture images is greater than or equal to the second number threshold, the head-mounted display device deletes a preset gesture image with the first response function label from the second data set, and adds the first gesture image with the first response function label to the second data set, wherein the deleted preset gesture image with the first response function label can be randomly selected or can be the one with the least similarity to the first gesture image. Through the above method, the storage space occupied by the second data set can be limited, and if the preset gesture image with the least similarity to the first gesture image is deleted from the second data set, it is equivalent to removing noise data in the second data set, which is conducive to improving the accuracy of the gesture classification model.
[0226] In the embodiment of the present application, the method for determining the second similarity is similar to the method for determining the first similarity, and will not be described in detail herein.
[0227] Through the above method, when the gestures made by the user are not standard, or when the texture information of the user's hand changes as the user ages, the gesture classification model can still perform accurate recognition.
[0228] In some embodiments, the head-mounted display device can directly add the first gesture image with the first username label and the first iris image with the first username label to the third data set, and use the updated third data set to train the second identity re-identification model.
[0229] In some embodiments, the head display device may also determine at least one preset gesture image with the first user name tag from the third data set, and use the determined preset gesture image with the first user name tag as the third candidate gesture image. Then, a third similarity between the at least one third candidate gesture image and the first gesture image is determined, and the following possible implementations may be performed according to the third similarity:
[0230] In some embodiments, when the third similarity is less than the fifth threshold, the head display device adds the first gesture image with the first username label and the first iris image with the first username label to the third data set, and uses the updated third data set to train the second identity re-identification model. When the third similarity is greater than or equal to the fifth threshold, the head display device does not perform other processing.
[0231] In some embodiments, when the third similarity is less than the fifth threshold and greater than the sixth threshold, the head display device adds the first gesture image with the first user name label and the first iris image with the first user name label to the third data set, and uses the updated third data set to train the second identity re-identification model. When the third similarity is greater than or equal to the fifth threshold, or when the third similarity is less than or equal to the sixth threshold, the head display device does not perform other processing. The sixth threshold is less than the fifth threshold.
[0232] The above method of obtaining the updated third data set is conducive to improving the accuracy of the second identity re-identification model. The specific analysis process can refer to the method of obtaining the updated first data set, which will not be described in detail here.
[0233] In some embodiments, when the third similarity is less than the fifth threshold, or when the third similarity is less than the fifth threshold and greater than the sixth threshold, before the head-mounted display device adds the first gesture image with the first username tag and the first iris image with the first username tag to the third data set, the head-mounted display device may determine whether the number of third candidate gesture images is less than the third number threshold; if the number of third candidate gesture images is less than the third number threshold, the head-mounted display device adds the first gesture image with the first username tag and the first iris image with the first username tag to the third data set; if the number of third candidate gesture images is greater than or equal to the third number threshold, the head-mounted display device deletes a preset gesture image with the first username tag and the corresponding preset iris image with the first username tag from the third data set, and adds the first gesture image with the first username tag and the first iris image with the first username tag to the third data set, wherein the deleted preset gesture image with the first username tag may be randomly selected or may have the smallest similarity with the first gesture image. Through the above method, the storage space occupied by the third data set can be limited. Moreover, if the preset gesture image with the least similarity to the first gesture image is deleted from the third data set, it is equivalent to removing the noise data in the third data set, which is beneficial to improving the accuracy of the second identity re-identification model.
[0234] In some embodiments, the head display device may also determine at least one preset iris image with the first user name tag from the third data set, and use the determined preset iris image with the first user name tag as the fourth candidate gesture image. Then, a fourth similarity between the at least one fourth candidate gesture image and the first gesture image is determined, and the following possible implementations may be performed according to the fourth similarity:
[0235] In some embodiments, when the fourth similarity is less than the seventh threshold, the head display device adds the first gesture image with the first username label and the first iris image with the first username label to the third data set, and uses the updated third data set to train the second identity re-identification model. When the fourth similarity is greater than or equal to the seventh threshold, the head display device does not perform other processing.
[0236] In some embodiments, when the fourth similarity is less than the seventh threshold and greater than the eighth threshold, the head display device adds the first gesture image with the first user name label and the first iris image with the first user name label to the third data set, and uses the updated third data set to train the second identity re-identification model. When the fourth similarity is greater than or equal to the seventh threshold, or when the fourth similarity is less than or equal to the eighth threshold, the head display device does not perform other processing. The eighth threshold is less than the seventh threshold.
[0237] In some embodiments, the head display device can also make a judgment based on the third candidate gesture image and the fourth candidate gesture image at the same time. Specifically, when the head display device determines that the third similarity is less than the fifth threshold and greater than the sixth threshold, and the fourth similarity is less than the seventh threshold and greater than the eighth threshold, the first gesture image with the first user name tag and the first iris image with the first user name tag are added to the third data set, and the updated third data set is used to update the second identity re-identification model. In other cases, the head display device does not perform other processing.
[0238] In the embodiment of the present application, the method for determining the third similarity and the fourth similarity is similar to the method for determining the first similarity, and will not be described in detail herein.
[0239] Through the above method, when the gestures made by the user are not standard, or when the texture information of the user's hand changes as the user ages, or when the user's iris changes, the second identity re-identification model can still perform accurate identification.
[0240] In an embodiment of the present application, the head display device can determine the first application space and the first response function based on the fifth preset feature.
[0241] In some embodiments, the head display device uses a machine learning method to extract features from the first gesture image to obtain a fifth feature, and then calculates the similarity between the fifth feature and at least one fifth preset feature. For any fifth preset feature, if the similarity between the fifth preset feature and the fifth feature is greater than a ninth threshold, the fifth preset feature is used as a candidate feature. The candidate feature with the highest similarity is determined from at least one candidate feature, and based on the fourth association relationship, the preset gesture image corresponding to the candidate feature with the highest similarity is determined. Based on the first association relationship or the third association relationship, the preset user name corresponding to the preset gesture image is determined, and the preset user name is used as the first user name, wherein the application space corresponding to the first user name is the first application space. Based on the second association relationship, the preset response function corresponding to the preset gesture image is determined, and the preset response function is used as the first response function.
[0242] It should be understood that the fifth feature may include at least one of the following: gesture joints, hand texture information. The fifth feature corresponds to the fifth preset feature. For example, if the fifth preset feature is a gesture joint, the fifth feature is also a gesture joint.
[0243] In some embodiments, the head display device uses a machine learning method to extract features from the first iris image to obtain a sixth feature, and then calculates the similarity between the sixth feature and at least one sixth preset feature. For any sixth preset feature, if the similarity between the sixth preset feature and the sixth feature is greater than a tenth threshold, the sixth preset feature is used as a candidate feature. The candidate feature with the highest similarity is determined from at least one candidate feature, and based on the fifth association relationship, the preset iris image corresponding to the candidate feature with the highest similarity is determined. Based on the third association relationship, the preset user name and the preset gesture image corresponding to the preset iris image are determined, and the preset user name is used as the first user name, wherein the application space corresponding to the first user name is the first application space. Based on the second association relationship, the preset response function corresponding to the preset gesture image is determined, and the preset response function is used as the first response function.
[0244] It should be understood that the sixth feature may include at least one of the following: gesture joints, hand texture information. The sixth feature corresponds to the sixth preset feature. For example, if the sixth preset feature is a gesture joint, the sixth feature is also a gesture joint.
[0245] In some embodiments, the head display device may also determine the first application space and the first response function based on the first gesture image and the first iris image. The specific method can be obtained by combining the above contents, which will not be described in detail here.
[0246] It should be understood that the machine learning method used by the head-mounted display device to obtain the fifth preset feature is the same as the machine learning method used by the head-mounted display device to obtain the fifth feature; the machine learning method used by the head-mounted display device to obtain the sixth preset feature is the same as the machine learning method used by the head-mounted display device to obtain the sixth feature.
[0247] In the above method, the head-mounted display device does not need to store the identity re-identification model (including the first identity re-identification model and / or the second identity re-identification model) and the gesture classification model, which saves the storage space of the head-mounted display device. In addition, the above method directly performs similarity calculation based on the features, which can conveniently determine the first application space and the first response function, simplifying the complexity of the process.
[0248] In some embodiments, when a first user is using a head-mounted display device, if the head-mounted display device obtains both a first gesture image of the first user and a second gesture image of the second user, since the first user is using the head-mounted display device, the head-mounted display device can obtain the iris image of the first user, i.e., the first iris image, and the head-mounted display device determines the corresponding first application space and the first response function based on the first gesture image and the first iris image, rather than determining the corresponding second application space and the second response function based on the second gesture image of the second user.
[0249] S1302: The head display device executes a first response function in the first application space.
[0250] The head display device starts a first application space corresponding to the first user name, and executes a first response function in the first application space.
[0251] Take the "Open Gallery" function as an example to illustrate the gesture for opening the gallery. Figure 6 As shown, if the first user makes a Fig.14 The gesture shown ( Figure 6 Gestures and Fig.14 ), the head display device obtains the gesture including Fig.14 , determining a first user name corresponding to the first gesture image, and an “open gallery” function corresponding to the first gesture image, wherein the application space corresponding to the first user name is the first application space. The head display device executes the “open gallery” function in the first application space.
[0252] In the above method, the first user can access the application software in the first application space, and at the same time, the first application space can only be accessed by the first user, which can effectively protect the privacy of the first user. Since different users correspond to different application spaces, the isolation of different application spaces can be achieved, thereby protecting the privacy of different users.
[0253] In the embodiment of the present application, when the first user uses a new head-mounted display device, the first user can directly transfer the information in the first application space in the original head-mounted display device to the new head-mounted display device, and the information in the first application space can be pictures, address books, call records, application software browsing records, etc. In addition, the information in the first application space can also include the associations related to the first user from the first association to the fifth association, as well as the first identity re-identification model, the second identity re-identification model, and the gesture classification model.
[0254] It is understood that when the original head-mounted display device transfers the information in the first application space to the new head-mounted display device, the original head-mounted display device can be triggered based on the first gesture image, wherein the first gesture image includes the transfer gesture; it can also be triggered based on a virtual key on the virtual keyboard.
[0255] Optionally, if the information in the first application space in the original head-mounted display device is stored in the original head-mounted display device, when the original head-mounted display device transfers the information in the first application space to the new head-mounted display device, the original head-mounted display device can also be connected to the new head-mounted display device, wherein the original head-mounted display device and the new head-mounted display device can be connected by wire or by wireless. If the original head-mounted display device and the new head-mounted display device are connected wirelessly, it can be understood that the wireless connection between the original head-mounted display device and the new head-mounted display device is achieved through the wireless communication module in the original head-mounted display device and the wireless communication module in the new head-mounted display device. For example, the original head-mounted display device and the new head-mounted display device can be wirelessly connected via WIFI or via Bluetooth.
[0256] If the information in the first application space in the original head display device is stored in the cloud server, when the original head display device transfers the information in the first application space to the new head display device, the original head display device is connected to the cloud server, and the new head display device is connected to the cloud server. Among them, the above connection methods are all wireless connections. The original head display device can be connected to the new head display device, or it can not be connected to the new head display device, which is not limited here.
[0257] By using the above method, when a user uses a new head display device, the user can transfer the information in the application space related to the user, and the information transfer can be realized conveniently and quickly. Since only the information in the application space related to the user is transferred, and the information in the application space related to other users is not transferred, information leakage can be avoided.
[0258] In the above embodiments provided by the present application, the method provided by the embodiment of the present application is introduced from the perspective of the head display device as the execution subject. In order to realize the various functions in the method provided by the above embodiments of the present application, the head display device may include a hardware structure and / or a software module, and realize the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.
[0259] For example, when hardware is used for implementation, the hardware implementation of the head display device can refer to Fig.15 and its related description.
[0260] See also Fig.15, the head display device may include: one or more processors 1502; a memory 1503; one or more applications (not shown); and one or more computer programs 1504, and the above-mentioned components may be connected via one or more communication buses 1501. The one or more computer programs 1504 are stored in the above-mentioned memory 1503 and are configured to be executed by the one or more processors 1502, and the one or more computer programs 1504 include instructions, and the above-mentioned instructions may be used to execute the method in any of the above-mentioned embodiments.
[0261] An embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on a head-mounted display device, the head-mounted display device executes the above-mentioned related method steps to implement the information interaction method of the head-mounted display device in the above-mentioned embodiment.
[0262] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the information interaction method of the head display device in the above-mentioned embodiment.
[0263] Among them, the head display device, computer storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.
[0264] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0265] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0266] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0267] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0268] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0269] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An information interaction method, applied to a head-mounted display device, characterized in that: The method comprises: Acquire a first gesture image of a first user, and determine a first application space and a first response function corresponding to the first gesture image, where the first application space is an application space allocated by a head-mounted display device to the first user, and the first response function is a response made by the head-mounted display device to the first gesture image; In the first application space, the first response function is executed.
2. The method according to claim 1, characterized in that The determining a first application space corresponding to the first gesture image includes: The first gesture image is input into a first identity re-identification model to obtain a first user name, and an application space corresponding to the first user name is used as the first application space, wherein the first identity re-identification model is obtained by training a first data set, and the first data set includes preset gesture images with preset user name labels.
3. The method according to claim 1, characterized in that The determining a first response function corresponding to the first gesture image includes: The first gesture image is input into a gesture classification model to obtain the first response function, wherein the gesture classification model is obtained by training with a second data set, and the second data set includes preset gesture images with preset response function labels.
4. The method according to claim 1, characterized in that After obtaining the first gesture image of the first user, the method further includes: Acquire a first iris image of the first user; The determining a first application space corresponding to the first gesture image includes: The first gesture image and the first iris image are input into a second identity re-identification model to obtain a first user name, and the application space corresponding to the first user name is used as the first application space. The second identity re-identification model is obtained by training a third data set, and the third data set includes a preset gesture image with a preset user name label, and a preset iris image with a preset user name label.
5. The method according to claim 2, characterized in that Also includes: The first gesture image with the first user name label is added to the first data set, and the first identity re-identification model is trained using the updated first data set.
6. The method according to claim 5, characterized in that The step of adding the first gesture image with the first user name label to the first data set includes: Determine at least one preset gesture image with the first user name label from the first data set; Determining a first similarity between the at least one preset gesture image with the first user name tag and the first gesture image; When the first similarity is smaller than a first threshold and larger than a second threshold, the first gesture image with the first user name label is added to the first data set.
7. The method according to claim 3, characterized in that Also includes: The first gesture image with the first response function label is added to the second data set, and the gesture classification model is trained using the updated second data set.
8. The method according to claim 7, characterized in that The step of adding the first gesture image with the first response function label to the second data set includes: Determining at least one preset gesture image with the first response function label from the second data set; determining a second similarity between the at least one preset gesture image with the first response function label and the first gesture image; When the second similarity is less than a third threshold and greater than a fourth threshold, the first gesture image with the first response function label is added to the second data set.
9. The method according to claim 4, characterized in that Also includes: The first gesture image with the first user name label and the first iris image with the first user name label are added to the third data set, and the second identity re-identification model is trained using the updated third data set.
10. The method according to claim 9, characterized in that The step of adding the first gesture image with the first user name label to the third data set includes: Determine at least one preset gesture image with the first user name label from the third data set; determining a third similarity between the at least one preset gesture image with the first user name tag and the first gesture image; When the third similarity is smaller than a fifth threshold and larger than a sixth threshold, the first gesture image with the first user name tag is added to the third data set.
11. A head mounted display device, characterized in that: including a processor and a memory; The processor is used to execute the computer program or instructions stored in the memory, so that the head display device implements the method described in any one of claims 1-10.
12. A computer-readable storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is executed on a computer, the computer-readable storage medium enables the computer-readable storage medium to implement the method according to any one of claims 1 to 10.
