Gesture recognition method, head-mounted wearable device, and storage medium

Abstract

The present application provides a gesture recognition method, a head-mounted wearable device, and a storage medium. The head-mounted wearable device comprises at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera. The method comprises: acquiring first image data under test, the first image data under test comprising at least two images collected by at least two cameras at a first moment; inputting the first image data under test into a keypoint detection model to obtain a first group of hand 3D keypoint coordinates, wherein the keypoint detection model is trained on the basis of at least one piece of sample image data, and each piece of sample image data comprises at least two preset gesture images marked with the hand 3D keypoint coordinates; and on the basis of the first group of hand 3D keypoint coordinates, recognizing a gesture type corresponding to the first image data under test. The present invention can improve the accuracy of gesture recognition in a complex scenario.

Description

A gesture recognition method, head-mounted wearable device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 15, 2023, with application number 202310713412.9 and application name “A gesture recognition method, head-mounted wearable device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of electronic technology, and in particular to a gesture recognition method, a head-mounted wearable device, and a storage medium. Background Art

[0004] With the advancement of virtual reality technology, head-mounted wearable devices are developing rapidly. Gesture interaction is becoming increasingly important when users use head-mounted wearable devices. However, the environment is often complex and changeable. Head-mounted wearable devices that only use grayscale or color cameras to capture images for gesture interaction are not sufficient for normal use in low-light environments or other complex environments. This results in a low success rate for gesture recognition when head-mounted wearable devices recognize user gestures based on captured images.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide a gesture recognition method, a head-mounted wearable device, and a storage medium to improve the gesture recognition success rate of the head-mounted wearable device.

[0007] In a first aspect, the present application provides a gesture recognition method for a head-mounted wearable device, wherein the head-mounted wearable device includes at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera. The method comprises: the head-mounted wearable device obtains first image data to be tested, wherein the first image data to be tested includes at least two images captured by at least two cameras at a first moment. The head-mounted wearable device inputs the first image data to be tested into a key point detection model to obtain a first set of 3D hand key point coordinates, wherein the key point detection model is trained based on at least one sample image data, each sample image data includes at least two preset gesture images labeled with 3D hand key point coordinates, and the at least two preset gesture images are captured by different types of cameras. Then, the head-mounted wearable device identifies the gesture type corresponding to the first image data to be tested based on the first set of 3D hand key point coordinates.

[0008] In this method, since the first image data to be tested includes images captured by multiple cameras, even if the first image data to be tested is in a complex scene or a scene with low light, sufficient information can be obtained for gesture recognition, thereby improving the accuracy of gesture recognition in complex scenes.

[0009] In some embodiments, each of the at least two preset gesture images is further annotated with a camera status value, that is, each sample image data includes at least two preset gesture images annotated with hand 3D key point coordinates and camera status values. Inputting the first image data to be tested into the key point detection model to obtain a first set of hand 3D key point coordinates includes: inputting the first image data to be tested into the key point detection model to obtain a first set of hand 3D key point coordinates and a first set of camera status values ​​corresponding to the camera included in the head-mounted wearable device. On the one hand, the head-mounted wearable device can accurately identify the gesture type corresponding to the first image data to be tested based on the first set of hand 3D key point coordinates; on the other hand, the first set of camera status values ​​can be used to reasonably use and adjust the status of the camera included in the head-mounted wearable device at the second moment.

[0010] In some embodiments, the method further includes: if the camera state of each camera included in the head-mounted wearable device does not match the first set of camera state values, then the head-mounted wearable device adjusts the state of the camera that does not match the first set of camera state values ​​in the head-mounted wearable device based on the first set of camera state values. This embodiment provides a method for facilitating adjustment of the head-mounted wearable device.

[0011] In some embodiments, after the head-mounted wearable device adjusts the state of the camera that does not match the first set of camera state values ​​based on the first set of camera state values, the method further includes: the head-mounted wearable device obtains second image data to be tested, the second image data to be tested includes an image captured by the camera in the head-mounted wearable device that is in an on state at a second moment, and the second moment is the next moment after the first moment. The head-mounted wearable device inputs the second image data to be tested into a key point detection model to obtain a second set of hand key point coordinates, and identifies the gesture type corresponding to the second image data to be tested based on the second set of hand key point coordinates. Since the second image data to be tested includes images captured by multiple cameras, even if the first image data to be tested is in a complex scene or a scene with low light, sufficient information can be obtained for gesture recognition, thereby improving the accuracy of gesture recognition in complex scenes.

[0012] In some embodiments, the method further includes: if a camera included in the head-mounted wearable device is in a turned-off state, turning on the turned-off camera when the head-mounted wearable device determines that a trigger condition has been met; the trigger condition includes at least one of the following: a change in the usage scenario of the head-mounted wearable device; a change in ambient brightness exceeding a first threshold; a change in hand movement speed exceeding a second threshold; or a change in gesture complexity. In this embodiment, when the trigger condition is met, all turned-off cameras can be triggered to turn on, allowing the head-mounted wearable device to dynamically adjust the camera status based on the actual scenario and effectively utilize each camera.

[0013] In a second aspect, the present application provides a gesture recognition method, which is applied to a head-mounted wearable device, wherein the head-mounted wearable device includes at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera. The method includes: obtaining third image data to be tested, the third image data to be tested including at least one grayscale image captured by the grayscale camera, at least one color image captured by the color camera, a depth image captured by the depth camera, and an event image captured by the event camera; segmenting the first image in the third image data to be tested to obtain N first sub-images, where the first image is a hand image of any one of at least one grayscale image and at least one color image, where N is an integer greater than 1; for each first sub-image in the N first sub-images, Image, executing: determining a second sub-image having a positional relationship with the first sub-image from each second image in at least one second image, the at least one second image including at least one image other than the first image in the third image data to be tested; inputting the first sub-image and / or at least one second sub-image into a key point detection model to obtain an output first sub-image corresponding to a third set of 3D hand key point coordinates, the key point detection model being trained based on at least one sample image data, each sample image data including at least two preset gesture images marked with 3D hand key point coordinates, and L being an integer greater than 1; identifying the gesture type corresponding to the third image data to be tested based on the third set of 3D hand key point coordinates corresponding to the N first sub-images respectively.

[0014] In this method, because the third image data to be tested includes images captured by multiple cameras, sufficient information can be obtained for gesture recognition even in complex scenes or scenes with low lighting, thereby improving the accuracy of gesture recognition in complex scenes. Furthermore, by performing 3D key point recognition on image regions within the third image data, accurate recognition can be achieved by region.

[0015] In some embodiments, before determining a second sub-image having a positional relationship with a first sub-image from each of the at least one second image, the method further includes: inputting each of the N first sub-images into a key point detection model to obtain the coordinates of the key points included in each first sub-image; determining a key point importance index corresponding to each first sub-image based on the key point coordinates included in each of the N first sub-images; and determining at least one second image from the third image data to be tested based on the key point importance indexes corresponding to the N first sub-images. In this embodiment, based on the importance of key points in different regions, key regions for gesture recognition can be accurately identified, and then corresponding images for inferring 3D key points can be determined, effectively utilizing computing resources.

[0016] In some embodiments, the at least one second image includes a depth image and an event image. In this embodiment, images for inferring 3D hand key points are obtained from the depth image and the event image in a region-by-region manner. This allows for more accurate coordinates of the 3D hand key points compared to images obtained from grayscale or color images.

[0017] In some embodiments, segmenting the first image in the third image data to obtain N first sub-images includes segmenting the first image in the third image data to obtain N equal parts to obtain N first sub-images; or segmenting the first image in the third image data to obtain N first sub-images according to N hand regions. This embodiment provides a specific method for accurately identifying 3D key points by region, which can improve the flexibility of the gesture recognition method.

[0018] In some embodiments, before segmenting the first image in the third image data to be tested to obtain N first sub-images, the method further includes: determining that the third image data to be tested includes a hand. In this embodiment, a method for performing 3D key point recognition by region is provided.

[0019] In a third aspect, the present application provides a gesture recognition method for a head-mounted wearable device, wherein the head-mounted wearable device includes at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera. The method includes: obtaining fourth image data to be tested, the fourth image data to be tested including at least two images captured by at least two cameras; determining a gesture level corresponding to the fourth image data to be tested; based on the gesture level, determining K images corresponding to the gesture level from the at least two images, where K is an integer greater than 1; inputting the K images into a key point detection model to obtain a fourth set of 3D hand key point coordinates as output, wherein the key point detection model is obtained by training at least one sample image data, each sample image data including at least two preset gesture images labeled with 3D hand key point coordinates, where L is an integer less than or equal to N; and identifying the gesture type corresponding to the fourth image data to be tested based on the fourth set of 3D hand key point coordinates.

[0020] In this method, because the fourth image data to be tested includes images captured by multiple cameras, sufficient information can be obtained for gesture recognition, even in complex scenes or scenes with low lighting, thereby improving gesture recognition accuracy in complex scenes. Furthermore, by determining the images used to infer 3D key points of the hand based on different gesture levels, the computing power of the head-mounted wearable device can be effectively utilized.

[0021] In some embodiments, the gesture level is determined based on gesture complexity or hand movement speed.

[0022] In a fourth aspect, the present application further provides a head-mounted wearable device, which can implement the method implemented in any of the first to third aspects above, or implement the method in any possible implementation of any of the first to third aspects. The head-mounted wearable device can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more units or modules corresponding to the above functions.

[0023] In a fifth aspect, embodiments of the present application further provide a head-mounted wearable device, comprising modules / units for executing the method implemented in any one of the first to third aspects, or for implementing the method in any possible implementation manner of any one of the first to third aspects. These modules / units may be implemented in hardware, or the corresponding software implementation may be executed in hardware.

[0024] In a sixth aspect, an embodiment of the present application provides a head-mounted wearable device, comprising a memory, a processor, and a display screen; wherein the display screen is used to display a user interface, and the processor is used to execute a computer program or instruction stored in the memory, so that the head-mounted wearable device implements the method implemented in any one of the first to third aspects above, or implements the method in any possible implementation manner of any one of the first to third aspects.

[0025] In the seventh aspect, a computer-readable storage medium is also provided in an embodiment of the present application, and the computer-readable storage medium includes a computer program. When the computer program is run on a head-mounted wearable device, the computer-readable storage medium implements the method implemented in any one of the first to third aspects above, or implements the method in any possible implementation manner of any one of the first to third aspects.

[0026] In an eighth aspect, an embodiment of the present application further provides a computer program product. When the computer program product is run on a head-mounted wearable device, the head-mounted wearable device implements the method implemented in any one of the first to third aspects above, or implements the method in any possible implementation manner of any one of the first to third aspects.

[0027] It should be understood that the beneficial effects of the fourth to eighth aspects of this application can be found in the relevant descriptions of the first to third aspects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a product schematic diagram of a head-mounted wearable device provided in an embodiment of the present application;

[0029] FIG2 is a schematic diagram of the hardware structure of a head-mounted wearable device provided in an embodiment of the present application;

[0030] FIG3 is a schematic diagram of a software structure of a head-mounted wearable device provided in an embodiment of the present application;

[0031] FIG4A is a schematic diagram of an architecture provided in an embodiment of the present application;

[0032] 4B to 4D are schematic diagrams of gestures provided in embodiments of the present application;

[0033] FIG4E is a schematic diagram of another architecture provided in an embodiment of the present application;

[0034] FIG5 is a schematic diagram of a key point detection model training process according to an embodiment of the present application;

[0035] FIG6 is a flowchart of a gesture recognition method provided in an embodiment of the present application;

[0036] FIG7 is a schematic diagram of key points of the hand provided by this application;

[0037] FIG8 is a flowchart of another gesture recognition method provided by an embodiment of the present application;

[0038] FIG9 is a flowchart of another gesture recognition method provided by an embodiment of the present application;

[0039] 10A to 10C are a set of image schematic diagrams provided in an embodiment of the present application;

[0040] 11A to 11E are another set of image schematic diagrams provided in an embodiment of the present application;

[0041] 12A to 12C are schematic diagrams of a process for detecting key points by region according to an embodiment of the present application;

[0042] FIG13 is a schematic diagram of a process for detecting key points by finger position according to an embodiment of the present application;

[0043] FIG14 is a flowchart of a gesture recognition method provided in an embodiment of the present application;

[0044] FIG15 is a schematic structural diagram of a head-mounted wearable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0046] As used in the specification and appended claims of this application, the singular expressions "a", "an", "said", "above", "the", and "this" are intended to include expressions such as "one or more", unless the context clearly indicates otherwise. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two, or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0047] References to "some embodiments" and the like in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0048] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0049] This application provides a gesture recognition method, a head-mounted wearable device, and a storage medium for improving the success rate of gesture recognition in the head-mounted wearable device. The method and the head-mounted wearable device are based on the same technical concept. Since the method and the head-mounted wearable device solve similar problems, the implementation of the head-mounted wearable device and the method can refer to each other, and any repetitions will not be repeated.

[0050] In the solution provided in the embodiment of the present application, the head-mounted wearable device includes a device capable of realizing data processing functions (such as a processor, or an application processor, or an image processor, or other processors). The exemplary embodiments of the head-mounted wearable device include but are not limited to a device equipped with Or devices with other operating systems. The above-mentioned head-mounted wearable device can also be other portable devices, such as a laptop computer with a touch-sensitive surface (such as a touch panel). The embodiments of the present application do not limit the specific form of the head-mounted wearable device.

[0051] Referring to Figure 1, the head-mounted wearable device can be the head-mounted wearable device 100 shown in Figure 1. Based on the head-mounted wearable device 100 shown in Figure 1, the head-mounted wearable device can also have other variant structural forms, which are not limited here. The head-mounted wearable device 100 can be smart glasses, such as AR glasses, VR glasses, etc. In the embodiment of the present application, the head-mounted wearable device 100 can be controlled by the user through the handle 200 in Figure 1, or can be directly controlled by the user through the fingers, or can be controlled through other devices, which are not limited here.

[0052] FIG2 is a schematic diagram of the hardware structure of a head-mounted wearable device 100 provided in an embodiment of the present application. Based on the structure shown in FIG2 , other variant structures may also exist. As shown in FIG2 , the head-mounted wearable device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include one or more of the following: a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0053] The following is a detailed introduction to the components of the head-mounted wearable device 100 shown in FIG. 2 .

[0054] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. In some embodiments, the head-mounted wearable device 100 may also include one or more processors 110. The processor is the nerve center and command center of the head-mounted wearable device 300. The processor may generate an operation control signal based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0055] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory, thereby avoiding repeated accesses and reducing processor 110 latency, thereby improving system efficiency.

[0056] In some embodiments, the processor 110 may include one or more interfaces. For example, the interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface. It will be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation of the head-mounted wearable device 100.

[0057] The charging management module 140 is configured to receive charging input from a charger. The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The wireless communication function of the head-mounted wearable device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, and a baseband processor.

[0058] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the head-mounted wearable device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0059] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the head-mounted wearable device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0060] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0061] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., applied to the head-mounted wearable device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0062] In some embodiments, antenna 1 of the head-mounted wearable device 100 is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, so that the head-mounted wearable device 100 can communicate with the network and other devices through wireless communication technology.

[0063] The head-mounted wearable device 100 implements display functions through a GPU, a display screen 194, and an application processor. The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the head-mounted wearable device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0064] In some embodiments, the head-mounted wearable device 100 may include at least one camera 193, and image acquisition can be achieved through the camera 193.

[0065] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the head-mounted wearable device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0066] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system and an application required for at least one function. In addition, the internal memory 121 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the head-mounted wearable device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0067] The head-mounted wearable device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0068] The head-mounted wearable device 100 can receive input from the button 190 and generate key signal input related to user settings and function control of the head-mounted wearable device 100. For example, the button 190 may include a power button, a volume up button, a volume down button, etc. set on the right side of the head-mounted wearable device.

[0069] Although not shown in FIG2 , the head-mounted wearable device 100 may further include a Bluetooth device, a positioning device, a flashlight, a micro-projection device, a near field communication (NFC) device, etc., which are not described in detail here.

[0070] It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the head-mounted wearable device 100. In other embodiments of the present application, the head-mounted wearable device 100 may include more or fewer components than those shown in FIG2 , or may combine or separate certain components, or arrange the components differently. The components shown in FIG2 may be implemented in hardware, software, or a combination of software and hardware.

[0071] The following embodiments can all be implemented in the head-mounted wearable device 100 having the above hardware structure.

[0072] The software system of the head-mounted wearable device can adopt a layered architecture, event-driven architecture, micro-core architecture, micro-service architecture, or cloud architecture. system) as an example to illustrate the software structure of the head-mounted wearable device.

[0073] Figure 3 exemplarily shows a schematic diagram of the software structure of a head-mounted wearable device provided in an embodiment of the present application. The layered architecture divides the software system into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, namely, the application layer (applications), the application framework layer (framework, FWK), the Android runtime (Android runtime) and the system library, and the kernel layer. The embodiment of the present application does not limit the layering of the software structure of the head-mounted wearable device.

[0074] As shown in FIG3 , the application layer may include a series of applications, including phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0075] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0076] As shown in FIG3 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0077] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0078] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0079] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0080] The phone manager is used to provide communication functions for the head-mounted display device 100, such as the management of call status (including answering, hanging up, etc.).

[0081] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, etc.

[0082] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0083] The Android Runtime consists of core libraries and a virtual machine (VM). The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one for Java-based functions and the other for the Android core library. The application layer and application framework layer run in the VM. The VM executes Java files from the application layer and application framework layer as binary files. The VM is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0084] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0085] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0086] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0087] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0088] A 2D graphics engine is a drawing engine for 2D drawings.

[0089] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, sensor driver, audio driver, and microphone driver.

[0090] The gesture recognition method provided in the embodiments of the present application can be applied to a variety of scenarios.

[0091] For example, when a user wears a head-mounted wearable device, the communication function of the mobile terminal can be controlled by recognizing the user's gestures. In this scenario, the user's gesture type and the response function can be bound together, for example, the "five-finger open" gesture type is bound to the call answering function, and the "V-shaped" gesture type is bound to the call rejection function. For example, taking the mobile terminal as a mobile phone 300, the system architecture shown in Figure 4A includes a head-mounted wearable device 100 and a mobile phone 300. When the mobile phone 300 receives an incoming call prompt, the head-mounted wearable device 100 displays the incoming call interface. If the user makes a "five-finger open" gesture as shown in Figure 4B, the head-mounted wearable device recognizes that the current user gesture type is a "five-finger open" gesture type, and then controls the mobile phone 300 to execute the incoming call answering function, so that the user can implement the incoming call answering function without removing the head-mounted wearable device 100; if the user makes a "V-shaped" gesture as shown in Figure 4C, the head-mounted wearable device 100 recognizes that the current user gesture type is a "V-shaped" gesture type, and then controls the mobile phone 300 to execute the incoming call rejection function, so that the user can implement the incoming call rejection function without removing the head-mounted wearable device.

[0092] The gesture recognition method provided in this application can also be applied to other scenarios. For example, a head-mounted wearable device can control its own functions by recognizing user gestures. For example, a user can use the head-mounted wearable device to recognize the type of user gesture and then control the head-mounted wearable device to open the installed software. For example, the "fist" gesture type is bound to the function of opening the gallery. When the user makes the "fist" gesture as shown in Figure 4D, the head-mounted wearable device recognizes that the current user gesture type is the "fist" gesture type, and the head-mounted wearable device opens the gallery.

[0093] In addition, the gesture recognition method provided by the present application can also be applied to other scenarios, such as in a smart home scenario, where the user can use a head-mounted wearable device to identify the type of user gesture, and then send a control instruction to a smart home device (such as a smart TV, table lamp, etc.) in the smart home scenario. For example, taking the smart home device as a smart TV 400 as an example, the system architecture shown in FIG4E includes a head-mounted wearable device 100 and a smart TV 400. For example, the "five-finger open" gesture type is bound to the function of turning on the smart TV, and the "fist" gesture type is bound to the function of turning off the smart TV. If the user makes the "five-finger open" gesture as shown in FIG4B, the head-mounted wearable device recognizes that the type of the user gesture is the "five-finger open" gesture type, then the head-mounted wearable device controls the smart TV to turn on; if the user makes the "fist" gesture as shown in FIG4D, the head-mounted wearable device recognizes that the type of the user gesture is the "fist" gesture type, then the head-mounted wearable device controls the smart TV to turn off. The head-mounted wearable device provided by the present application can also be applied to scenarios for controlling other devices, which will not be described in detail here.

[0094] In the embodiment of the present application, the head-mounted wearable device may include at least two cameras. The gesture recognition method provided in the embodiment of the present application does not limit the specific number of cameras included in the head-mounted wearable device, and will not be repeated hereafter. In the following embodiment, a head-mounted wearable device including four cameras, namely a grayscale camera, a color camera, a depth camera and an event camera, is used as an example for introduction. When using the head-mounted wearable device to capture images, these cameras can be all started or partially started. The head-mounted wearable device can capture the image to be detected through at least one of the grayscale camera, the color camera, the depth camera and the event camera, and then infer the 3D key point coordinates of the hand corresponding to the image to be detected through the key point detection model, and then identify the corresponding gesture type based on the 3D key point coordinates of the hand.

[0095] The grayscale camera involved in the embodiments of the present application can be, for example, a binocular grayscale camera, a four-eye grayscale camera, a six-eye grayscale camera, or a grayscale camera with more eyes. Among them, the binocular grayscale camera can capture two grayscale images, the four-eye binocular grayscale camera can capture two grayscale images, and the six-eye grayscale camera can capture two grayscale images. This will not be repeated later.

[0096] The color camera involved in the embodiments of the present application can be, for example, a binocular color camera, a four-eye color camera, a six-eye color camera, or a color camera with more eyes. Among them, the binocular color camera can capture two color images, the four-eye binocular color camera can capture two color images, and the six-eye color camera can capture two color images. This will not be repeated below.

[0097] The depth camera involved in the embodiments of the present application can be, for example, an indirect time-of-flight (IToF) camera, a direct time-of-flight (DToF) camera, a structured light camera, etc., which will not be described in detail below.

[0098] Before performing gesture recognition on the image to be detected, the process of training the key point detection model is explained.

[0099] Referring to FIG5 , there is shown an exemplary diagram of a key point detection model training process provided by an embodiment of the present application, which is performed by a head-mounted wearable device. The key point detection model training process includes the following steps:

[0100] Step 501: The head-mounted wearable device obtains at least one sample image data.

[0101] Each sample image data includes a plurality of preset gesture images captured by different types of cameras.

[0102] The specific implementation of obtaining sample image data is introduced below.

[0103] For example, a head-mounted wearable device includes a grayscale camera, a color camera, a depth camera, and an event camera. All four cameras are started. The four cameras can obtain a sufficient amount of original image data in a variety of complex scenarios in advance. Taking scene one in which the lights are turned off indoors and the ambient light is weak as an example, the four cameras respectively capture images of both hands in scene one. Each capture in scene one can obtain an original image data, including six images, namely, two darker grayscale images P1 and grayscale image P2 captured by the binocular grayscale camera, two darker color images P3 and P4 captured by the binocular color camera, a depth image P5 containing depth information of both hands in front of the camera captured by the depth camera, and an event image P6 containing information on changes in brightness during the extension of both hands.

[0104] In some embodiments, the user can label the hand 3D key point coordinates on the six images included in each original image data obtained by shooting scene one. The six images labeled with the hand 3D key point coordinates can be used as a sample image data. This sample image data includes: a grayscale image P1 labeled with the hand 3D key point coordinates X1, a grayscale image P2 labeled with the hand 3D key point coordinates X2, a color image P3 labeled with the hand 3D key point coordinates X3, a color image P4 labeled with the hand 3D key point coordinates X4, a depth image P5 labeled with the hand 3D key point coordinates X5, and an event image P6 labeled with the hand 3D key point coordinates X6.

[0105] In some embodiments, in addition to labeling the 3D hand keypoint coordinates for each of the six images included in the raw image data captured in Scene 1, the user can also label the camera status for these six images. For example, in Scene 1, the lights are off indoors and the ambient light is weak. The two grayscale images captured by the binocular grayscale camera contain very little information and are almost unusable. Based on these two grayscale images, the camera status V1 can be labeled as 0. Similarly, the two color images captured by the binocular color camera also contain very little information. Based on these two color images, the camera status V2 can be labeled as 0. The depth image captured by the depth camera in Scene 1 includes depth information, which is useful for gesture recognition solutions. Based on these depth images, the camera status V3 can be labeled as 1. The event camera is not affected by light during shooting, and the event image captured includes event information. Based on these event images, the camera status V4 can be labeled as 1. Therefore, the six images labeled with the hand's 3D keypoint coordinates and camera status can be used as sample image data. This sample image data includes: grayscale image P1 labeled with V1 = 0 and hand's 3D keypoint coordinates X1, grayscale image P2 labeled with V1 = 0 and hand's 3D keypoint coordinates X2, color image P3 labeled with V2 = 0 and hand's 3D keypoint coordinates X3, color image P4 labeled with V2 = 0 and hand's 3D keypoint coordinates X4, depth image P5 labeled with V3 = 1 and hand's 3D keypoint coordinates X5, and event image P6 labeled with V4 = 1 and hand's 3D keypoint coordinates X6. A sample image data can be regarded as a training sample and used as model input.

[0106] In an embodiment of the present application, the head-mounted wearable device can take multiple shots in scene one, and each shot can obtain a sample image data. Each sample image data includes six images taken by four cameras. For example, 100 sample image data are taken in scene one, and the camera status annotations of six images in these sample image data can be consistent with the above example. If the user gestures corresponding to these 100 sample image data are different, the corresponding annotated hand 3D key point coordinates will also be different.

[0107] To improve the accuracy of the keypoint detection model in gesture recognition, collect as much sample image data as possible in a variety of complex scenarios when training the keypoint detection model. For example, collect sample image data in indoor scenes with open windows and strong ambient light, or in outdoor scenes with strong ambient light.

[0108] In step 502, the head-mounted wearable device inputs at least one sample image data into a key point detection model to be trained to obtain a prediction result corresponding to each sample image data.

[0109] Taking the six images with marked hand 3D key point coordinates as a sample image data as an example, the prediction results corresponding to the sample image data include: the predicted values ​​of the hand 3D key points corresponding to the six images, that is, the predicted value X1′ of the hand 3D key point coordinates corresponding to the grayscale image P1, the predicted value X2′ of the hand 3D key point coordinates corresponding to the grayscale image P2, the predicted value X3′ of the hand 3D key point coordinates corresponding to the color image P3, the predicted value X4′ of the hand 3D key point coordinates corresponding to the color image P4, the predicted value X5′ of the hand 3D key point coordinates corresponding to the depth image P5, and the predicted value X6′ of the hand 3D key point coordinates corresponding to the event image P6.

[0110] Taking the six images marked with the 3D key point coordinates of the hand and the camera status as a sample image data as an example, the prediction results corresponding to the sample image data include: the predicted values ​​of the camera status and the predicted values ​​of the 3D key points of the hand corresponding to the six images, that is, the predicted value V1′ of the camera status and the predicted value X1′ of the 3D key point coordinates of the hand corresponding to the grayscale image P1, the predicted value V1′ of the camera status and the predicted value X2′ of the 3D key point coordinates of the hand corresponding to the grayscale image P2, the predicted value V2′ of the camera status and the predicted value X3′ of the 3D key point coordinates of the hand corresponding to the color image P3, the predicted value V2′ of the camera status and the predicted value X4′ of the 3D key point coordinates of the hand corresponding to the color image P4, the predicted value V3′ of the camera status and the predicted value X5′ of the 3D key point coordinates of the hand corresponding to the depth image P5, and the predicted value V4′ of the camera status and the predicted value X6′ of the 3D key point coordinates of the hand corresponding to the event image P6.

[0111] In step 503, the head-mounted wearable device determines a loss value between the predicted result and the true result of each sample image data according to the predicted result and the true result of each sample image data in the at least one sample image data.

[0112] Taking the aforementioned six images with labeled 3D hand keypoint coordinates as a sample image data set as an example, the loss value between the predicted result and the true result for each sample image data set can be determined based on the loss value W1 between the true value and the predicted value of the 3D hand keypoint coordinates corresponding to the sample image data. The true value of the 3D hand keypoint coordinates corresponding to the sample image data is the labeled 3D hand keypoint coordinates for each image included in the sample image data.

[0113] Here, the loss value W1 can be composed of the loss value W11 between the true value X1 and the predicted value X1′ of the hand 3D key point coordinates corresponding to the grayscale image P1, the loss value W12 between the true value X2 and the predicted value X2′ of the hand 3D key point coordinates corresponding to the grayscale image P2, the loss value W13 between the true value X3 and the predicted value X3′ of the hand 3D key point coordinates corresponding to the color image P3, the loss value W14 between the true value X4 and the predicted value X4′ of the hand 3D key point coordinates corresponding to the color image P4, the loss value W15 between the true value X5 and the predicted value X5′ of the hand 3D key point coordinates corresponding to the depth image P5, and the loss value W16 between the true value X6 and the predicted value X6′ of the hand 3D key point coordinates corresponding to the event image P6. Then, the average value of W11, W12, W13, W14, W15 and W16 can be used as the loss value W1 between the true value and the predicted value of the 3D key point coordinates of the hand corresponding to the sample image data, or W11, W12, W13, W14, W15 and W16 can be weighted and summed according to certain weights to obtain the loss value W1. This application does not limit this.

[0114] Taking the aforementioned six images with labeled 3D key point coordinates of the hand and camera status as a sample image data as an example, the loss value between the predicted result and the actual result of each sample image data can be determined based on the loss value W2 between the actual value and the predicted value of the camera status corresponding to the sample image data, and the loss value W1 between the actual value and the predicted value of the 3D key point coordinates of the hand corresponding to the sample image data. For example, the loss value corresponding to the sample image data is the sum of the loss value W2 and the loss value W1. For another example, the loss value corresponding to the sample image data is the average of the loss value W2 and the loss value W1. For another example, the weight coefficients of the loss value W2 and the loss value W1 are set respectively, and the first product of the loss value W2 and its weight coefficient, and the second product of the loss value W1 and its weight coefficient are determined respectively. The loss value corresponding to the sample image data is the sum of the first product and the second product. Among them, the loss value W1 can be referred to the aforementioned example and will not be repeated here.

[0115] Here, the loss value W2 can be composed of the loss value W21 between the true value V1 and the predicted value V1′ of the camera state corresponding to the grayscale image P1, the loss value W22 between the true value V1 and the predicted value V1′ of the camera state corresponding to the grayscale image P2, the loss value W23 between the true value V2 and the predicted value V2′ of the camera state corresponding to the color image P3, the loss value W24 between the true value V2 and the predicted value V2′ of the camera state corresponding to the color image P4, the loss value W25 between the true value V3 and the predicted value V3′ of the camera state corresponding to the depth image P5, and the loss value W26 between the true value V4 and the predicted value V4′ of the camera state corresponding to the event image P6. Then, the average value of W21, W22, W23, W24, W25 and W26 can be used as the loss value W2 between the true value and the predicted value of the camera state corresponding to the sample image data, or W21, W22, W23, W24, W25 and W26 can be weighted and summed according to certain weights to obtain the loss value W2. This application does not limit this.

[0116] In step 504, the head-mounted wearable device updates the weights of the key point detection model according to the loss value between the predicted result and the actual result of each sample image data in at least one sample image data until a trained key point detection model is obtained, and the training is completed.

[0117] The process of step 504 can be completed in the head-mounted wearable device or in other devices. When step 504 is completed in other devices, the other devices can transmit the model parameters corresponding to the trained key point detection model to the head-mounted wearable device so that the head-mounted wearable device updates the key point detection model.

[0118] This step 504 can be implemented in the following manner: determine whether the loss value between the predicted result and the true result of the sample image data is less than a preset expected value; if so, it means that the predicted result is close enough to the true result, and the trained key point detection model is obtained, and the training is completed; if not, the loss value of one or more sample image data is reversely input into the key point detection model being trained, and the weight of the key point detection model is updated to obtain an updated key point detection model, and then continue to repeat the above steps 502 to step 504, by continuously returning the loss value, continuously updating the weight of the key point detection model, and continuously obtaining the loss value of the sample image data through the key point detection model after updating the weight, until the loss value of the sample image data in the test set is less than the preset expected value, and the training is completed.

[0119] In an embodiment of the present application, if several images with labeled 3D key point coordinates of the hand are used as sample image data, and a large amount of such sample image data is used as input for model training, the obtained key point detection model can be used to infer the 3D key point coordinates of the hand corresponding to the image data to be tested. For the sake of convenience of description, this key point detection model may be referred to as the first key point detection model below.

[0120] If several images with labeled 3D key point coordinates of the hand and camera status are used as sample image data, and a large number of such sample image data are used as input for model training, the obtained key point detection model can be used to infer the 3D key point coordinates of the hand and camera status corresponding to the image data to be tested. For the sake of convenience of description, this key point detection model can be referred to as the second key point detection model below.

[0121] After introducing the process of training the key point detection model, the gesture recognition process of this application is introduced in detail.

[0122] In some embodiments, when gesture recognition is required during the use of a head-mounted wearable device, the image data to be tested acquired through multiple cameras contains more information. Then, gesture recognition processing is performed on the currently acquired image data to be tested through the key point detection model trained as described above, which can improve the accuracy of gesture recognition in complex scenarios.

[0123] As shown in Figure 6, a flowchart of a gesture recognition method provided by an embodiment of the present application is exemplified. The method is applied to a head-mounted wearable device, which includes at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera. The gesture recognition method includes the following steps:

[0124] In step 601, the head-mounted wearable device obtains first image data to be tested, where the first image data to be tested includes images captured by the at least two cameras at a first moment.

[0125] In an embodiment of the present application, when a user uses a head-mounted wearable device, at least two cameras in the head-mounted wearable device are turned on, and the first image data to be tested includes images captured by all cameras in the head-mounted wearable device. Taking the head-mounted wearable device as an example, where the head-mounted wearable device includes a binocular grayscale camera, a binocular color camera, a depth camera, and an event camera, the first image data to be tested includes six images, namely, a first grayscale image and a second grayscale image captured by the binocular grayscale camera, a first color image and a second color image captured by the binocular color camera, a depth image captured by the depth camera, and an event image captured by the event camera.

[0126] In step 602, the head-mounted wearable device inputs the first image data to be tested into the trained key point detection model to obtain a first set of hand 3D key point coordinates output by the key point detection model.

[0127] The key point detection model in step 602 can be the first key point detection model or the second key point detection model. Taking the first key point detection model as an example, the first grayscale image, the second grayscale image, the first color image, the second color image, the depth image, and the event image are input into the first key point detection model, and a first set of 3D hand key point coordinates is output.

[0128] Exemplarily, the first set of hand 3D key point coordinates may include 42 key point coordinates, for example, the first 21 key point coordinates represent the key point coordinates of the left hand, and the last 21 key point coordinates represent the key point coordinates of the right hand. If the first image data to be tested includes both hands, the first set of hand 3D key points includes 42 hand key point coordinates. If the first image data to be tested includes a single hand, for example, only the left hand, then after the first image data to be tested is input into the first key point detection model, the output first set of hand 3D key points includes 42 key point coordinates, of which the last 21 key point coordinates are all (0, 0, 0), that is, the 21 key point coordinates of the right hand are all (0, 0, 0).

[0129] Taking the right hand as an example, the key point locations of a single hand are explained. For a single hand, there are 21 hand key points, and the specific locations can be numbered 0-20. As shown in Figure 7, the 21 hand key points include a palm base key point (key point 0) and 20 finger key points (key points 1-20). Each finger contains 4 key points. Among the 20 finger key points, key points 1, 5, 9, 13, and 17 are finger base key points, key points 2, 6, 10, 14, and 18 are proximal finger key points, key points 3, 7, 11, 15, and 19 are distal finger key points, and key points 4, 8, 12, 16, and 20 are fingertip key points.

[0130] In step 603, the head-mounted wearable device identifies the gesture type corresponding to the first image data to be tested based on the first set of 3D hand key point coordinates.

[0131] For example, the head-mounted wearable device can extract gesture feature parameters based on the first set of 3D hand key point coordinates, and then input the gesture feature parameters into a first gesture recognition model to output a gesture type. The first gesture recognition model can be trained using a large amount of gesture feature data labeled with preset gesture types.

[0132] For example, the head-mounted wearable device can directly input the first set of hand 3D key point coordinates into the second gesture recognition model to obtain an output gesture type. The second gesture recognition model can be trained using multiple sets of hand 3D key point coordinates labeled with preset gesture types.

[0133] Optionally, if the key point detection model in step 602 is the aforementioned second key point detection model, then in step 602, the head-mounted wearable device inputs the first image data to be tested into the trained second key point detection model. In addition to outputting the first set of 3D hand key point coordinates, the second key point detection model can also output a first set of camera status values. The first set of camera status values ​​can be used to adjust the state of the camera included in the head-mounted wearable device at a second moment, where the second moment is the next moment after the first moment. Through the camera status values ​​output by the second key point detection model, various cameras in the head-mounted wearable device are reasonably used. For example, the first set of camera status values ​​includes the camera status value V1 of the binocular grayscale camera, the camera status value V2 of the binocular color camera, the camera status value V3 of the depth camera, and the camera status value V4 of the event camera. The first set of camera state values ​​can reflect the environmental information at the first moment. For example, the first set of camera state values ​​includes V1, V2, V3, and V4, which are 0, 0, 1, and 1 respectively. Among them, V1 is 0 and V2 is 0, indicating that there is very little information in the images captured by the binocular grayscale camera and the binocular color camera, which is not helpful for gesture recognition. The binocular grayscale camera and the binocular grayscale camera can be turned off to reduce the power consumption of the head-mounted wearable device, and the depth camera and the event camera can be kept on. Then, at the next moment after the first moment, the depth camera and the event camera in the head-mounted wearable device will respectively capture images, while the binocular grayscale camera and the binocular grayscale camera will no longer capture images.

[0134] In some embodiments, when the key point detection model in step 602 is the second key point detection model, after step 603 , as shown in FIG. 8 , the gesture recognition method may further include steps 801 to 805 .

[0135] In step 801, the head-mounted wearable device controls the states of four cameras in the head-mounted wearable device according to a first set of camera state values ​​output by a key point detection model.

[0136] In one possible implementation, if the camera status of each camera included in the head-mounted wearable device does not match the first set of camera status values, the status of the cameras in the head-mounted wearable device that do not match the first set of camera status values ​​is adjusted according to the first set of camera status values.

[0137] For example, at the first moment, the binocular grayscale camera, binocular color camera, depth camera and event camera included in the head-mounted wearable device are all in the on state, and the first group of camera state values ​​output by the second key point detection model are 1, 0, 1, and 1, respectively, where 1 represents the on state and 0 represents the off state. According to the states of various cameras in the head-mounted wearable device at the first moment and the first group of camera state values, it can be seen that the binocular color camera in the head-mounted wearable device at the first moment does not match V2 of the first group of camera state values ​​of 0, then the binocular color camera is turned off, while the binocular grayscale camera, depth camera and event camera remain on, so that the binocular grayscale camera, depth camera and event camera can collect images at the second moment. For another example, at the first moment, the binocular grayscale camera, binocular color camera, depth camera and event camera included in the head-mounted wearable device are all in the on state, and the camera state value output by the second key point detection model is 0, 0, 0, 1, where 1 represents the on state and 0 represents the off state. According to the states of various cameras in the head-mounted wearable device at the first moment and the first set of camera state values, it can be seen that at the first moment, the binocular grayscale camera, binocular color camera and depth camera in the head-mounted wearable device do not match V1, V2 and V3 in the first set of camera state values ​​of 0, then the binocular grayscale camera, binocular color camera and depth camera are turned off, and the event camera remains on, so that at the second moment only the event camera can collect images.

[0138] In step 802, the head-mounted wearable device obtains second image data to be tested, where the second image data to be tested includes an image captured by a camera in an on state in the head-mounted wearable device at a second moment.

[0139] For example, taking the first group of camera state values ​​as 1, 0, 1, 1 as an example, at the second moment, the binocular grayscale camera, depth camera and event camera are in the on state, and the second image data to be tested includes two grayscale images P11 and P12 captured by the binocular grayscale camera, a depth image P13 captured by the depth camera, and an event image P14 captured by the event camera.

[0140] In step 803, the head-mounted wearable device inputs the second image data to be tested into the trained second key point detection model to obtain a second set of camera state values ​​and a second set of hand 3D key point coordinates output by the second key point detection model.

[0141] The second set of camera status values ​​may be used to adjust the status of a camera included in the head-mounted wearable device at a third moment, where the third moment is the next moment after the second moment.

[0142] In step 804 , the head-mounted wearable device identifies the gesture type corresponding to the second image data to be tested based on the second set of 3D hand key point coordinates.

[0143] The specific implementation of identifying the gesture type in step 804 can be found in the related description of the aforementioned step 603, which will not be repeated here.

[0144] In step 805 , the head-mounted wearable device controls the states of four cameras in the head-mounted wearable device according to the second set of camera state values ​​output by the key point detection model.

[0145] If the camera status of each camera included in the head-mounted wearable device does not match the second set of camera status values, the status of the camera in the head-mounted wearable device that does not match the second set of camera status values ​​is adjusted according to the second set of camera status values.

[0146] For example, the first group of camera state values ​​is 0, 0, 0, 1, that is, at the second moment, the binocular grayscale camera, binocular color camera and depth camera included in the head-mounted wearable device are all in the off state, and the event camera is in the on state. The second group of camera state values ​​is 0, 1, 1, 1. At this time, the binocular color camera and the depth camera are both in the off state, which does not match V2 and V3 in the second group of camera state values. Therefore, the binocular color camera and the depth camera are turned on, that is, at the third moment, the binocular grayscale camera is in the off state, and the binocular color camera, depth camera and event camera are all in the on state. In this way, at the third moment, the binocular color camera, depth camera and event camera respectively capture images.

[0147] Through the above embodiment, after the head-mounted wearable device collects the image data to be tested at the current moment, the image data to be tested is input into a set of hand 3D key point coordinates and a set of camera status values ​​that can be output by the second key point detection model, wherein the set of hand 3D key point coordinates output by the model can identify the gesture type at the current moment, and the set of camera status values ​​output by the model can determine the state of the camera used to collect the image data to be tested at the next moment. Since the second key point detection model is trained through sample images collected in various scenes, the set of camera status values ​​output by the model is also combined with environmental information, thereby realizing the dynamic collection of the image data to be tested in combination with the environment.

[0148] In any of the above embodiments, if a camera included in the head-mounted wearable device is in an off state, when the head-mounted wearable device determines that a trigger condition is satisfied, the off camera is turned on, that is, all cameras are turned on. Satisfying the trigger condition includes at least one of the following:

[0149] (1) The usage scenario of the head-mounted wearable device changes;

[0150] For example, the usage scenario of the head-mounted wearable device switches from playing badminton to watching a movie, or from watching a movie to a gaming scenario.

[0151] (2) Changes in ambient brightness;

[0152] For example, if the change value of the ambient brightness exceeds the first threshold, it indicates that the ambient brightness has changed.

[0153] (3) changes in hand movement speed;

[0154] For example, if the change in the hand movement speed exceeds the second threshold, it indicates that the hand movement speed has changed.

[0155] (4) The complexity of gestures changes.

[0156] Through the above embodiment, if the trigger condition is met, the camera that is in the off state is turned on, so that all cameras included in the head-mounted wearable device are in the on state when collecting the image data to be tested at the next moment.

[0157] Referring to FIG9 , there is shown a flow chart illustrating another method for gesture recognition provided by an embodiment of the present application. The method is applied to a head-mounted wearable device, which includes at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera. The gesture recognition method includes the following steps:

[0158] In step 901, the head-mounted wearable device obtains third image data to be tested, where the third image data to be tested includes images captured by the at least two cameras.

[0159] Taking the head-mounted wearable device including a binocular grayscale camera, a binocular color camera, a depth camera and an event camera as an example, the third image data to be tested includes a first grayscale image (for example, called image P1) and a second grayscale image (for example, called image P2) captured by the binocular grayscale camera, a first color image (for example, called image P3) and a second color image (for example, called image P4) captured by the binocular color camera, a depth image (for example, called image P5) captured by the depth camera, and an event image (for example, called image P6) captured by the event camera.

[0160] In step 902 , the head-mounted wearable device performs hand detection on the third image data to be tested.

[0161] Step 902 is an optional step.

[0162] In some embodiments, the head-mounted wearable device can perform hand detection on the first grayscale image and / or the second grayscale image. If a hand is detected, step 903 is continued; otherwise, step 903 and its subsequent steps are not performed. For example, taking image P1 as an example, the head-mounted wearable device can perform hand detection on image P1. If a hand is detected in image P1, step 903 is continued; otherwise, step 903 and its subsequent steps are not performed.

[0163] Taking the first grayscale image captured by a binocular grayscale camera as an example, the first grayscale image is, for example, image P1 as shown in FIG10A . The head-mounted wearable device performs hand detection on image P1, identifies that image P1 includes a single hand, and then determines the coordinates of the four vertices A1, A2, A3, and A4 of the minimum rectangular area that includes the hand in image P1. The four vertices of the minimum rectangular area can also be mapped to the coordinate system of other cameras, and the mapping area of ​​the minimum rectangular area in the image captured by the corresponding camera (such as the first color image, the second color image, the depth image, the event image, etc.) can be obtained. This mapping area is the minimum rectangular area that includes the hand in the other image. For example, the minimum rectangular area in image P1 shown in Figure 10A is mapped to image P3 captured by the binocular color camera, and the coordinates of each vertex A1, A2, A3 and A4 of the minimum rectangular area shown in Figure 10A are respectively mapped to the coordinates in the coordinate system of image P3, resulting in a mapping area surrounded by four vertices A1′, A2′, A3′ and A4′ as shown in Figure 10B. The image corresponding to the mapping area is called a hand image, such as image P3′ shown in 10B.

[0164] It should be understood that the above-mentioned minimum rectangular area can also be replaced by the minimum square area including the hand, or other shapes including the hand, which is not limited here.

[0165] In step 903, the head-mounted wearable device segments the first image in the third image data to be tested to obtain N first sub-images, where N is an integer greater than 1.

[0166] The first image may be a hand image of any one of a first grayscale image, a second grayscale image, a first color image, and a second color image.

[0167] In the embodiment of the present application, the above step 903 can be implemented in multiple ways.

[0168] In implementation A1, the first image in the third image data to be tested is divided into N equal parts to obtain N first sub-images.

[0169] Taking N as 4 and the first image being the image P3′ shown in FIG. 10B as an example, as shown in FIG. 10C , the image P3′ is divided into four first sub-images, namely image P31′, image P32′, image P33′, and image P34′.

[0170] In implementation A2, the first image in the third image data to be tested is segmented according to the regions where the N hand parts are located to obtain N first sub-images.

[0171] Taking the first image as image P3′ shown in Figure 10B as an example, a single hand includes five hand parts, namely the thumb, index finger, middle finger, ring finger, and little finger. According to these five hand parts, the image P3′ can be divided according to the areas where each hand part is located, and the image P311′ corresponding to the index finger area as shown in Figure 11A, the image P312′ corresponding to the middle finger area as shown in Figure 11B, the image P313′ corresponding to the ring finger area as shown in Figure 11C, the image P314′ corresponding to the little finger area as shown in Figure 11D, and the image P315′ corresponding to the thumb area as shown in Figure 11E are obtained.

[0172] In step 904, the head-mounted wearable device determines, for each of the N first sub-images, a second sub-image having a positional relationship with the first sub-image from the third image data to be tested, and inputs the first sub-image and / or the at least one second sub-image into the key point detection model to obtain a third set of hand 3D key point coordinates.

[0173] In some embodiments, determining, from the third image data to be tested, a second sub-image having a positional relationship with each first sub-image in the N first sub-images in step 904 can be implemented as follows: for each first sub-image in the N first sub-images, determining, from each second image in the at least one second image, a second sub-image having a positional relationship with the first sub-image. There are multiple possible implementations of the at least one second image.

[0174] Implementation method B1: N first sub-images are respectively input into the key point detection model to obtain the coordinates of the key points included in each first sub-image; then, based on the key point coordinates included in each first sub-image in the N first sub-images, the key point importance index corresponding to each first sub-image is determined; based on the key point importance index corresponding to the N first sub-images, at least one second image is determined from the third image data to be tested.

[0175] For example, taking N first sub-images as shown in FIG10C , images P31′, P32′, P33′, and P34′, respectively, as an example, images P31′, P32′, P33′, and P34′ are input into the key point detection model, and the output 3D hand key point coordinates corresponding to image P31′, image P32′, image P33′, and image P34′ are obtained. Based on the 3D hand key point coordinates corresponding to images P31′, P32′, P33′, and P34′, the 3D hand key points contained in images P31′, P32′, P33′, and P34′ can be determined. Then, based on the 3D hand key points contained in images P31′, P32′, P33′, and P34′, the key point importance indices corresponding to images P31′, P32′, P33′, and P34′ can be determined. The finger locations included in each first sub-image can be determined based on the 3D hand key points contained in each first sub-image, and the key point importance indices for each first sub-image can then be determined based on the order of key point importance.

[0176] For example, the order of keypoint importance from most important to least important is: index fingertip; finger tips; finger joints; wrist. The greater the keypoint's importance, the higher the keypoint importance index can be set. The coordinates of the hand's 3D keypoints are determined based on the principle that "the greater the keypoint importance (index fingertip > finger tips > finger joints > wrist), the more complex the solution used to calculate the more accurate 3D hand keypoints." For example, the key point importance index is set to 1, 2, or 3. When the key point importance index is 3, the mapping area image corresponding to the first sub-image in the image captured by the binocular color camera, the mapping area image corresponding to the first sub-image in the image captured by the depth camera, and the mapping area image corresponding to the first sub-image in the image captured by the event camera are input into the key point detection model to infer the 3D key point coordinates of the hand included in the first sub-image. When the key point importance index is 2, the mapping area image corresponding to the first sub-image in the image captured by the binocular color camera and the mapping area image corresponding to the first sub-image in the image captured by the depth camera are input into the key point detection model to infer the 3D key point coordinates of the hand included in the first sub-image. When the key point importance index is 1, the mapping area image corresponding to the first sub-image in the image captured by the binocular color camera is input into the key point detection model to infer the 3D key point coordinates of the hand included in the first sub-image. This is explained below with reference to Figures 12A to 12C.

[0177] (1) The index finger tip is included in image P31′. According to the key point importance principle, its corresponding key point importance index is determined to be 3. Images P3, P4, P5 and P6 can be used to determine the 3D key points corresponding to the hand part included in image P31′.

[0178] As shown in Figure 12A, based on the coordinates of the four vertices of image P31′, image P31′ is mapped to images P4, P5, and P6, respectively. This results in image P41 in image P4 that corresponds to image P31′, image P51 in image P5 that corresponds to image P31′, and image P61 in image P6 that corresponds to image P31′. Images P31′, P41, P51, and P61 are then input into a keypoint detection model to obtain 3D keypoints corresponding to the hand parts included in image P31′.

[0179] (2) The image P32′ includes the tip of the index finger, and the key point importance index corresponding to the image P32′ is 1. For example, the image P32′ can be used to determine the 3D key points corresponding to the hand part included in the image P32′.

[0180] (3) Image P33′ includes fingertips, finger joints, and finger tips such as the middle finger and the ring finger. According to the key point importance principle, the corresponding key point importance index is determined to be 2. Images P3, P4, and P5 can be used to determine the 3D key points corresponding to the hand parts included in image P33′.

[0181] As shown in Figure 12B, based on the coordinates of the four vertices of image P33′, image P33′ is mapped to images P4 and P5, respectively, resulting in image P43 in image P4 that corresponds to image P33′, and image P53 in image P5 that corresponds to image P33′. Images P33′, P43, and P53 are then input into the keypoint detection model to obtain 3D keypoints corresponding to the hand parts included in image P33′.

[0182] (4) Image P34′ includes the wrist, thumb, etc., and the corresponding key point importance index is determined to be 2 according to the key point importance principle. Images P3, P4, and P5 can be used to determine the 3D key points corresponding to the hand parts included in image P34′.

[0183] As shown in Figure 12C, based on the coordinates of the four vertices of image P34′, image P34′ is mapped to images P4 and P5, respectively, resulting in image P44 in image P4 that corresponds to image P34′, and image P54 in image P5 that corresponds to image P34′. Images P34′, P44, and P54 are then input into the keypoint detection model, yielding the 3D keypoints corresponding to the hand parts included in the output image P34′.

[0184] Through the above example, we can divide the image into regions and then use different strategies to determine the 3D key points in each region based on the importance of the key points, thereby obtaining more accurate 3D key points of the hand. Moreover, for regions with less important key points, fewer images are used to calculate the 3D key points in the corresponding area, which can save computing power and device power consumption.

[0185] In implementation B2, the at least one second image includes a depth image and an event image.

[0186] Based on this implementation B2, for each first sub-image among the N first sub-images, a sub-depth image that has a positional correspondence with the first sub-image can be determined from the depth image, and a sub-event image that has a positional correspondence with the first sub-image can be determined from the event image; then the sub-depth image that has a positional correspondence with the first sub-image and the sub-event image that has a positional correspondence with the first sub-image are input into the key point detection model to obtain a third set of 3D key point coordinates of the hand.

[0187] For example, N first sub-images are respectively image P311′ corresponding to the index finger region as shown in FIG11A , image P312′ corresponding to the middle finger region as shown in FIG11B , image P313′ corresponding to the ring finger region as shown in FIG11C , image P314′ corresponding to the little finger region as shown in FIG11D , and image P315′ corresponding to the thumb region as shown in FIG11E . The segmented images P311′, P312′, P313′, P314′, and P315′ are mapped to images P5 and P6 respectively, and sub-images of each finger region are mapped to images P5 and P6 respectively. For each finger region, the coordinates of the finger region are mapped to the sub-images of images P5 and P6 and input into the key point detection model to output the 3D key points included in the finger region.

[0188] For example, taking image P311′ corresponding to the index finger region as an example, as shown in Figure 13, the coordinates corresponding to image P311′ are mapped to images P5 and P6, respectively, resulting in image P511 corresponding to the coordinates of image P311′ in image P5, and image P611 corresponding to the coordinates of image P311′ in image P6. Images P511 and P611 are then input into the keypoint detection model to obtain the 3D keypoint coordinates of the index finger.

[0189] Taking image P312′ corresponding to the middle finger region as an example, the coordinates corresponding to image P312′ are mapped to images P5 and P6, respectively, resulting in image P512 corresponding to the coordinates of image P312′ in image P5 and image P612 corresponding to the coordinates of image P312′ in image P6. Images P512 and P612 are then input into the keypoint detection model to obtain the 3D keypoint coordinates of the middle finger.

[0190] Similarly, the 3D key point coordinates of the ring finger, little finger, and thumb are obtained respectively.

[0191] Through the above embodiment, the image is divided into regions according to each finger, and then the images captured by the depth camera and the event camera are used in each region to obtain more accurate 3D key points of each finger.

[0192] In step 905 , the head-mounted wearable device identifies the gesture type corresponding to the third image data to be tested based on the third set of hand 3D key point coordinates corresponding to the N first sub-images.

[0193] The specific implementation of identifying the gesture type in step 905 can be found in the related description of the aforementioned step 603, which will not be repeated here.

[0194] In this embodiment of the present application, because the third image data to be tested includes images captured by multiple cameras, sufficient information can be obtained for gesture recognition even in complex scenes or scenes with low lighting, thereby improving the accuracy of gesture recognition in complex scenes. Furthermore, by performing 3D key point recognition on image regions within the third image data, accurate recognition can be achieved by region.

[0195] In some embodiments, the gesture recognition method can be implemented based on the complexity of the gesture or the speed of the hand movement.

[0196] Referring to FIG14 , there is shown a flow chart illustrating another method for gesture recognition provided by an embodiment of the present application. The method is applied to a head-mounted wearable device, which includes at least two of the following cameras: a binocular grayscale camera, a binocular color camera, a depth camera, and an event camera. The gesture recognition method includes the following steps:

[0197] In step 1401, the head-mounted wearable device obtains fourth image data to be tested, where the fourth image data to be tested includes at least two images captured by at least two cameras.

[0198] For example, the fourth image data to be tested includes a first grayscale image and a second grayscale image captured by a binocular grayscale camera, a first color image and a second color image captured by a binocular color camera, a depth image captured by a depth camera, and an event image captured by an event camera.

[0199] The specific implementation of step 1401 can refer to the relevant description of step 801 above, which will not be repeated here.

[0200] In step 1402, the head-mounted wearable device performs hand detection on the fourth image data to be tested.

[0201] The specific implementation of step 1401 can refer to the relevant description of step 802 above, which will not be repeated here.

[0202] Step 1403: The head-mounted wearable device determines a gesture level corresponding to the fourth image data to be tested.

[0203] The gesture level corresponding to the fourth image data to be tested can be determined based on the gesture complexity or hand movement speed corresponding to the fourth image data to be tested. For example, gesture complexity can be divided into M levels, with each level corresponding to a gesture level. The more complex the gesture, the higher the gesture complexity level, and an image containing more information can be selected for inferring the 3D key points of the hand. Similarly, for example, hand movement speed can be divided into M levels. The faster the hand movement speed, the higher the level, and an image containing more information can be selected for inferring the 3D key points of the hand.

[0204] In step 1404, the head-mounted wearable device determines, based on the gesture level, K images corresponding to the gesture level from the at least two images, where K is an integer greater than 1.

[0205] For example, when M is 4, when the level is 1, the corresponding K images include images captured by the binocular grayscale camera, that is, the first grayscale image (for example, called image P1) and the second grayscale image (for example, called image P2); when the level is 2, the corresponding K images include images captured by the binocular grayscale camera and the binocular color camera, that is, the first grayscale image, the second grayscale image, the first color image (for example, called image P3) and the second color image (for example, called image P4); when the level is 3, the corresponding K images include images captured by the binocular grayscale camera, the binocular color camera and the depth camera, that is, the first grayscale image, the second grayscale image, the first color image, the second color image (for example, called image P4) and the depth image (for example, called image P5); when the level is 4, the corresponding K images include images in the binocular grayscale camera, the binocular color camera, the depth camera and the event camera, that is, the first grayscale image, the second grayscale image, the first color image, the second color image, the depth image (for example, called image P5) and the event image (for example, called image P6).

[0206] In some embodiments, a third image in the fourth image data to be tested may be first obtained. The third image may be any one of the first grayscale image, the second grayscale image, the first color image, and the second color image. A hand image including a hand is then determined in the first image. For example, the third image is image P1 as shown in FIG. 10A , and the hand image is image P1′. Then, based on the vertex coordinates of the hand image, the hand image is mapped to images corresponding to the gesture level. For example, image P1′ is mapped to the corresponding sub-image P2′ in image P2, to the corresponding sub-image P3′ in image P3, to the corresponding sub-image P4′ in image P4, to the corresponding sub-image P5′ in image P5, and to the corresponding sub-image P6′ in image P6. In this example, the K images include sub-image P1′, sub-image P2′, sub-image P3′, sub-image P4′, sub-image P5′, and sub-image P6′.

[0207] In step 1405, the head-mounted wearable device inputs the K images into a keypoint detection model to output a fourth set of 3D hand keypoint coordinates. The keypoint detection model is trained using at least one sample image data set, each of which includes at least two preset hand gesture images labeled with 3D hand keypoint coordinates.

[0208] In an example, taking the gesture complexity determined as 5 based on P1, the corresponding K images are image P1, image P2, image P3, image P4, image P5 and image P6. Image P1, image P2, image P3, image P4, image P5 and image P6 are input into the key point detection model to obtain the output of the fourth set of hand 3D key point coordinates.

[0209] Taking the gesture complexity of 5 determined according to P1 as an example, the corresponding K images are sub-image P1′, sub-image P2′, sub-image P3′, sub-image P4′, sub-image P5′ and sub-image P6′, and then sub-image P1′, sub-image P2′, sub-image P3′, sub-image P4′, sub-image P5′ and sub-image P6′ are input into the key point detection model to obtain the output 3D key point coordinates of the hand.

[0210] Step 1406 : Identify the gesture type corresponding to the fourth image data to be tested based on the fourth set of hand 3D key point coordinates.

[0211] The specific implementation of identifying the gesture type in step 1406 can be found in the related description of the aforementioned step 603, which will not be repeated here.

[0212] In the above example, because the fourth image data to be tested includes images captured by multiple cameras, even in complex scenes or low-light scenes, sufficient information can be obtained for gesture recognition, thereby improving gesture recognition accuracy in complex scenes. Furthermore, by determining the images used to infer 3D key points of the hand based on different gesture levels, the computing power resources of the head-mounted wearable device can be effectively utilized.

[0213] In the embodiments provided in the present application above, the method provided in the embodiments of the present application is introduced from the perspective of a head-mounted wearable device as the execution subject. In order to implement the various functions in the methods provided in the embodiments of the present application above, the head-mounted wearable device may include a hardware structure and / or a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0214] Exemplarily, when hardware implementation is adopted, the hardware implementation of the head-mounted wearable device can refer to Figure 15 and its related description.

[0215] 15 , the head-mounted display device may include: one or more processors 1502; a memory 1503; one or more application programs (not shown); and one or more computer programs 1504. The aforementioned components may be connected via one or more communication buses 1501. The one or more computer programs 1504 are stored in the memory 1503 and configured to be executed by the one or more processors 1502. The one or more computer programs 1504 include instructions, which may be used to execute the method in any of the aforementioned embodiments.

[0216] An embodiment of the present application also provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on a head-mounted wearable device, the head-mounted wearable device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0217] The embodiments of the present application also provide a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0218] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0219] Among them, the head-mounted wearable device, computer storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0220] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0222] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0223] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0224] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0225] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A gesture recognition method, applied to a head-mounted wearable device, characterized in that: The head-mounted wearable device includes at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera, and the method includes: Acquire first image data to be tested, where the first image data to be tested includes at least two images captured by the at least two cameras at a first moment; Inputting the first image data to be tested into a key point detection model to obtain a first set of 3D key point coordinates of the hand, wherein the key point detection model is obtained by training based on at least one sample image data, each of which includes at least two preset gesture images annotated with 3D key point coordinates of the hand, and the at least two preset gesture images are acquired by different types of cameras; According to the first set of hand 3D key point coordinates, a gesture type corresponding to the first image data to be tested is identified.

2. The method according to claim 1, characterized in that Each of the at least two preset gesture images is also marked with a camera state value; The step of inputting the first image data to be tested into a key point detection model to obtain a first set of hand 3D key point coordinates includes: The first image data to be tested is input into the key point detection model to obtain the first group of hand 3D key point coordinates and the first group of camera state values ​​corresponding to the camera included in the head-mounted wearable device.

3. The method according to claim 2, characterized in that The method further comprises: If the camera status of each camera included in the head-mounted wearable device does not match the first set of camera status values, the status of the cameras in the head-mounted wearable device that do not match the first set of camera status values ​​is adjusted according to the first set of camera status values.

4. The method according to claim 3, characterized in that After adjusting the states of cameras that do not match the first group of camera state values ​​according to the first group of camera state values, the method further includes: Acquire second image data to be tested, where the second image data to be tested includes an image captured by a camera in the head-mounted wearable device that is turned on at a second moment, where the second moment is a moment next to the first moment; Inputting the second image data to be tested into the key point detection model to obtain a second set of hand key point coordinates; According to the second set of hand key point coordinates, a gesture type corresponding to the second image data to be tested is identified.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: In a case where there is a camera in a turned-off state among the cameras included in the head-mounted wearable device, when the head-mounted wearable device determines that a trigger condition is met, turning on the camera in the turned-off state; The trigger condition includes at least one of the following: The usage scenario of the head-mounted wearable device is switched; The change value of the ambient brightness exceeds the first threshold; The change in hand movement speed exceeds a second threshold; The gesture complexity changes.

6. A gesture recognition method, applied to a head-mounted wearable device, characterized in that: The head-mounted wearable device includes at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera, and the method includes: Acquire third image data to be tested, wherein the third image data to be tested includes at least one grayscale image captured by the grayscale camera, at least one color image captured by the color camera, a depth image captured by the depth camera, and an event image captured by the event camera; Segmenting the first image in the third image data to be tested to obtain N first sub-images, where the first image is a hand image of any one of the at least one grayscale image and the at least one color image, and N is an integer greater than 1; For each first sub-image of the N first sub-images, executing: Determine a second sub-image having a positional relationship with the first sub-image from each second image of at least one second image, wherein the at least one second image includes at least one image in the third image data to be tested except the first image; Inputting the first sub-image and / or at least one second sub-image into a key point detection model to obtain a third set of 3D hand key point coordinates corresponding to the first sub-image, wherein the key point detection model is trained based on at least one sample image data, each of the sample image data includes at least two preset gesture images annotated with 3D hand key point coordinates, and L is an integer greater than 1; According to the third groups of hand 3D key point coordinates corresponding to the N first sub-images respectively, the gesture type corresponding to the third image data to be tested is identified.

7. The method according to claim 6, characterized in that Before determining, from each of the at least one second image, a second sub-image having a positional relationship with the first sub-image, the method further includes: Inputting the N first sub-images into a key point detection model respectively to obtain the coordinates of the key points included in each of the first sub-images; Determining a key point importance index corresponding to each first sub-image according to the key point coordinates included in each first sub-image of the N first sub-images; According to the key point importance indicators respectively corresponding to the N first sub-images, the at least one second image is determined from the third image data to be tested.

8. The method according to claim 6, characterized in that The at least one second image includes the depth image and the event image.

9. The method according to any one of claims 6 to 8, characterized in that: The step of segmenting the first image in the third image data to be tested to obtain N first sub-images includes: Divide the first image in the third image data to be tested into N equal parts to obtain the N first sub-images; or, The first image in the third image data to be tested is segmented according to the regions where the N hand parts are located to obtain the N first sub-images.

10. The method according to any one of claims 6 to 9, characterized in that: Before segmenting the first image in the third image data to be tested to obtain N first sub-images, the method further includes: It is determined that the third image data to be tested includes a hand.

11. A gesture recognition method, applied to a head-mounted wearable device, characterized in that: The head-mounted wearable device includes at least two of the following cameras: a grayscale camera, a color camera, a depth camera, and an event camera, and the method includes: Acquire fourth image data to be tested, wherein the fourth image data to be tested includes at least two images captured by the at least two cameras; Determining a gesture level corresponding to the fourth image data to be tested; According to the gesture level, determining K images corresponding to the gesture level from the at least two images, where K is an integer greater than 1; Input the K images into a key point detection model to obtain a fourth set of 3D hand key point coordinates as output, wherein the key point detection model is obtained by training with at least one sample image data, each of the sample image data includes at least two preset gesture images annotated with 3D hand key point coordinates, and L is an integer less than or equal to N; According to the fourth set of hand 3D key point coordinates, the gesture type corresponding to the fourth image data to be tested is identified.

12. The method according to claim 11, characterized in that The gesture level is determined according to gesture complexity or hand movement speed.

13. A head-mounted wearable device, characterized in that: Includes processor, memory and display screen; The display screen is used to display a user interface; The memory is used to store one or more computer programs, and when the computer programs are executed by the processor, the head-mounted wearable device executes the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a computer program, and when the computer program is executed on a head-mounted wearable device, the head-mounted wearable device executes the method according to any one of claims 1 to 12.

15. A computer program product, characterized in that The computer program product stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 12.

Images